[OE-core] [PATCH v2 1/7] lib/oe/package_manager: re-implement rpm feed signing

2017-08-11 Thread Markus Lehtonen
[YOCTO #11209] Signed-off-by: Markus Lehtonen <markus.lehto...@linux.intel.com> --- meta/lib/oe/package_manager.py | 14 +- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/meta/lib/oe/package_manager.py b/meta/lib/oe/package_manager.py index 518cf8dbe3..942f

Re: [OE-core] [PATCH 5/5] dnf: expand dnf selftest to test signed package feeds

2017-08-11 Thread Markus Lehtonen
On 11/08/2017, 14.11, "Alexander Kanavin" <alexander.kana...@linux.intel.com> wrote: On 08/11/2017 01:51 PM, Markus Lehtonen wrote: > --- a/meta-selftest/lib/oeqa/runtime/cases/dnf_runtime.py > +++ b/meta-selftest/lib/oeqa/runtime/cases/dnf_runtime.py >

[OE-core] [PATCH 4/5] oeqa: fix temp file handling in dnf package feed test

2017-08-11 Thread Markus Lehtonen
Prevent stale temp files and a possible (if unlikely) race in tempfile usage. Signed-off-by: Markus Lehtonen <markus.lehto...@linux.intel.com> --- meta-selftest/lib/oeqa/runtime/cases/dnf_runtime.py | 9 ++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/meta-selfte

[OE-core] [PATCH 5/5] dnf: expand dnf selftest to test signed package feeds

2017-08-11 Thread Markus Lehtonen
[YOCTO #12099] Signed-off-by: Markus Lehtonen <markus.lehto...@linux.intel.com> --- meta-selftest/files/signing/key.passphrase | 1 + meta-selftest/lib/oeqa/runtime/cases/dnf_runtime.py | 12 meta/lib/oeqa/selftest/cases/runtime_test.py| 13 - 3

[OE-core] [PATCH 2/5] dnf: rrecommend gnupg

2017-08-11 Thread Markus Lehtonen
This makes it possible to enable 'repo_gpgcheck' in dnf.conf. That is, do GPG signature check on repository metadata. Without gnupg dnf fails with "error: Invalid crypto engine." [YOCTO #11209] Signed-off-by: Markus Lehtonen <markus.lehto...@linux.intel.com> --- meta/reci

[OE-core] [PATCH 3/5] oeqa: fix dnf tests

2017-08-11 Thread Markus Lehtonen
Rename one dnf runtime test that it will recognized as a python module and thus also found by the oe test loader. Also, fix value of TEST_SUITES in dnf selftest so that all test dependencies are satisfied and the runtime test may be successfully run from there. Signed-off-by: Markus Lehtonen

[OE-core] [PATCH 1/5] lib/oe/package_manager: re-implement rpm feed signing

2017-08-11 Thread Markus Lehtonen
[YOCTO #11209] Signed-off-by: Markus Lehtonen <markus.lehto...@linux.intel.com> --- meta/lib/oe/package_manager.py | 14 +- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/meta/lib/oe/package_manager.py b/meta/lib/oe/package_manager.py index 518cf8dbe3..942f

[OE-core] [PATCH 0/5] Support signed RPM package feeds

2017-08-11 Thread Markus Lehtonen
ble in the git repository at: git://git.openembedded.org/openembedded-core-contrib marquiz/fixes-11209 http://cgit.openembedded.org/openembedded-core-contrib/log/?h=marquiz/fixes-11209 Markus Lehtonen (5): lib/oe/package_manager: re-implement rpm feed signing dnf: rrecommend gnupg o

[OE-core] [PATCH v4 0/8] support profile-optimized build for Python

2017-08-08 Thread Markus Lehtonen
-core-contrib/log/?h=marquiz/fixes-9338 Markus Lehtonen (8): python3-native: support profile optimized build python3: fix depends of python3-tests python3: add python3-profile-opt recipe devtools/images: add python-pgo-image python3: support profile optimized build python3: fix profile

[OE-core] [PATCH v3 4/8] devtools/images: add python-pgo-image

2017-05-18 Thread Markus Lehtonen
be specified with PYTHON3_PROFILE_TASK. The default profile task is pybench, although an example for using test.regrtest is included in the recipe. [YOCTO #9338] Signed-off-by: Markus Lehtonen <markus.lehto...@linux.intel.com> --- meta/recipes-devtools/images/python-pgo-image.b

[OE-core] [PATCH v3 1/8] python3-native: support profile optimized build

2017-05-18 Thread Markus Lehtonen
.conf, e.g. PYTHON3_NATIVE_PROFILE_TASK = "${S}/Tools/pybench/pybench.py -n 2 --with-gc --with-syscheck" [YOCTO #9338] Signed-off-by: Markus Lehtonen <markus.lehto...@linux.intel.com> --- meta/recipes-devtools/python/python3-native_3.5.2.bb | 9 + meta/recipes-devtools/python/python3/0

[OE-core] [PATCH v3 0/8] support profile-optimized build for Python

2017-05-18 Thread Markus Lehtonen
nce commit 138f3e3f81d866e8272b01dd88c164b367adc4cd: byacc: remove the recipe (2017-05-16 14:08:19 +0100) are available in the git repository at: git://git.openembedded.org/openembedded-core-contrib marquiz/fixes-9338 http://cgit.openembedded.org/openembedded-core-contrib/log/?h=marquiz/fixes-9338 Markus L

[OE-core] [PATCH v3 3/8] python3: add python3-profile-opt recipe

2017-05-18 Thread Markus Lehtonen
the target hardware. This patch enables the first step, i.e. building python with profile instrumentation enabled [YOCTO #9338] Signed-off-by: Markus Lehtonen <markus.lehto...@linux.intel.com> --- meta/recipes-devtools/python/python3-profile-opt | 1 + .../python/python3-profile-op

[OE-core] [PATCH v3 8/8] python3: remove two setup.py cross-compile hacks

2017-05-18 Thread Markus Lehtonen
en running "gcc -E -v". The second hack becomes useless after the first hack has been removed and we get the standard include and lib directories normally from gcc. Signed-off-by: Markus Lehtonen <markus.lehto...@linux.intel.com> --- .../python/python3-profile-opt_3.5.2.bb

[OE-core] [PATCH v3 6/8] python3: fix profile-optimized build of modules

2017-05-18 Thread Markus Lehtonen
Without this the pgo-related compiler flags are not used in cross-builds. [YOCTO #9338] Signed-off-by: Markus Lehtonen <markus.lehto...@linux.intel.com> --- ...-CFLAGS-for-extensions-when-cross-compili.patch | 56 ++ meta/recipes-devtools/python/python3_3.5.2.bb

[OE-core] [PATCH v3 5/8] python3: support profile optimized build

2017-05-18 Thread Markus Lehtonen
to specify PYTHON3_PROFILE_OPT = "1" in your local.conf, and, have profile data available in the location pointed to by PYTHON3_PROFILE_DIR. Profile data can be obtainen e.g. by running bitbake python-pgo-image -c profile3. [YOCTO #9338] Signed-off-by: Markus Lehtonen <

[OE-core] [PATCH v3 7/8] python3: add python3-tools subpackage

2017-05-18 Thread Markus Lehtonen
Useful in developing Python, e.g. in benchmarking. [YOCTO #9338] Signed-off-by: Markus Lehtonen <markus.lehto...@linux.intel.com> --- meta/recipes-devtools/python/python-3.5-manifest.inc | 8 +--- meta/recipes-devtools/python/python3_3.5.2.bb| 5 + 2 files changed, 10 inse

[OE-core] [PATCH v3 2/8] python3: fix depends of python3-tests

2017-05-18 Thread Markus Lehtonen
Similar to an earlier fix for Python 2.7. Make the tests subpackage depend on all modules as test.regrtest uses most (if not all) of them. Some tests also depend on libgcc so add that as a runtime dependency as well. [YOCTO #9338] Signed-off-by: Markus Lehtonen <markus.lehto...@linux.intel.

Re: [OE-core] [PATCH v2 00/20] support profile-optimized build for Python

2017-05-16 Thread Markus Lehtonen
On 16/05/2017, 16.45, "Richard Purdie" <richard.pur...@linuxfoundation.org> wrote: On Tue, 2017-05-16 at 16:18 +0300, Markus Lehtonen wrote: > This patchset makes it possible to make a PGO (profile-guided- > optimization) build of python. This version of

Re: [OE-core] [PATCH v2 00/20] support profile-optimized build for Python

2017-05-16 Thread Markus Lehtonen
On 16/05/2017, 16.23, "Alexander Kanavin" <alexander.kana...@linux.intel.com> wrote: On 05/16/2017 04:18 PM, Markus Lehtonen wrote: > This patchset makes it possible to make a PGO (profile-guided-optimization) > build of python. This version of the patchset is

[OE-core] [PATCH v2 13/20] python3: add python-profile-opt3 recipe

2017-05-16 Thread Markus Lehtonen
3. [YOCTO #9338] Signed-off-by: Markus Lehtonen <markus.lehto...@linux.intel.com> --- meta/recipes-devtools/python/python-profile-opt3 | 1 + .../python/python-profile-opt3_3.5.2.bb| 15 +++ ...efile-add-install_generate_profile-target.patch | 25 + ...name-libpyth

[OE-core] [PATCH v2 12/20] python3: fix depends of python-tests

2017-05-16 Thread Markus Lehtonen
Similar to an earlier fix for Python 2.7. Make the tests subpackage depend on all modules as test.regrtest uses most (if not all) of them. Some tests also depend on libgcc so add that as a runtime dependency as well. [YOCTO #9338] Signed-off-by: Markus Lehtonen <markus.lehto...@linux.intel.

[OE-core] [PATCH v2 10/20] python-pgo-image: enable sstate for do_profile

2017-05-16 Thread Markus Lehtonen
tbake python-pgo-image -c clean" (or otherwise wipe pre-existing profile data) before running "bitbake python-pgo-image -c profile". [YOCTO #9338] Signed-off-by: Markus Lehtonen <markus.lehto...@linux.intel.com> --- meta/recipes-devtools/images/python-pgo-image.bb | 15 +

[OE-core] [PATCH v2 19/20] python3: add python3-tools subpackage

2017-05-16 Thread Markus Lehtonen
[YOCTO #9338] Signed-off-by: Markus Lehtonen <markus.lehto...@linux.intel.com> --- meta/recipes-devtools/python/python-3.5-manifest.inc | 8 +--- meta/recipes-devtools/python/python3_3.5.2.bb| 5 + 2 files changed, 10 insertions(+), 3 deletions(-) diff --git a/meta/r

[OE-core] [PATCH v2 16/20] python3: support profile optimized build

2017-05-16 Thread Markus Lehtonen
e data can be obtainen e.g. by running bitbake python-pgo-image -c profile3. [YOCTO #9338] Signed-off-by: Markus Lehtonen <markus.lehto...@linux.intel.com> --- meta/recipes-devtools/python/python3_3.5.2.bb | 20 1 file changed, 20 insertions(+) diff --git a/meta/recipes

[OE-core] [PATCH v2 15/20] python3: remove two setup.py cross-compile hacks

2017-05-16 Thread Markus Lehtonen
en running "gcc -E -v". The second hack becomes useless after the first hack has been removed and we get the standard include and lib directories normally from gcc. [YOCTO #9338] Signed-off-by: Markus Lehtonen <markus.lehto...@linux.intel.com> --- .../python3-setup.py-no-host-

[OE-core] [PATCH v2 09/20] python-pgo-image: switch python default profile task to pybench

2017-05-16 Thread Markus Lehtonen
Switch to pybench in order to reduce the time taken by profiling. [YOCTO #9338] Signed-off-by: Markus Lehtonen <markus.lehto...@linux.intel.com> --- meta/recipes-devtools/images/python-pgo-image.bb | 18 ++ 1 file changed, 10 insertions(+), 8 deletions(-) diff --git

[OE-core] [PATCH v2 20/20] python-pgo-image: change python3 profile target to pybench

2017-05-16 Thread Markus Lehtonen
Running test.regrtest was still taking ridiculously long, i.e. around 4 hours in qemux86 on my desktop machine. Change to pybench which "only" takes around 55 minutes. [YOCTO #9338] Signed-off-by: Markus Lehtonen <markus.lehto...@linux.intel.com> --- meta/recipes-devtools/i

[OE-core] [PATCH v2 11/20] python3-native: support profile optimized build

2017-05-16 Thread Markus Lehtonen
This feature is virtually identical to that of python-native. Profile guided optimization is enabled by defining PYTHON3_NATIVE_PROFILE_OPT = "1" in local.conf. In addition, the profile task may be defined with PYTHON3_NATIVE_PROFILE_TASK. [YOCTO #9338] Signed-off-by: Marku

[OE-core] [PATCH v2 17/20] python3: fix profile-optimized build of modules

2017-05-16 Thread Markus Lehtonen
Without this the pgo-related compiler flags are not used in cross-builds. [YOCTO #9338] Signed-off-by: Markus Lehtonen <markus.lehto...@linux.intel.com> --- ...-CFLAGS-for-extensions-when-cross-compili.patch | 56 ++ meta/recipes-devtools/python/python3_3.5.2.bb

[OE-core] [PATCH v2 18/20] python-pgo-image: exclude tests from the python3 profile target

2017-05-16 Thread Markus Lehtonen
test_urllib2_localnet: 1276.8s test_itertools: 944.7s test_tuple: 847.7s test_trace: 756.7s test_tarfile: 716.7s test_unicodedata: 709.9s test_decimal: 615.1s [YOCTO #9338] Signed-off-by: Markus Lehtonen <markus.lehto...@linux.intel.com> --- meta/recipes-devtools/images/pyth

[OE-core] [PATCH v2 08/20] python: add python-tools subpackage

2017-05-16 Thread Markus Lehtonen
Useful in developing Python, e.g. in benchmarking. [YOCTO #9338] Signed-off-by: Markus Lehtonen <markus.lehto...@linux.intel.com> --- meta/recipes-devtools/python/python-2.7-manifest.inc | 8 +--- meta/recipes-devtools/python/python_2.7.13.bb| 5 + 2 files changed, 10 inse

[OE-core] [PATCH v2 14/20] python-pgo-image: profiling for python3

2017-05-16 Thread Markus Lehtonen
with PYTHON3_PROFILE_TASK. [YOCTO #9338] Signed-off-by: Markus Lehtonen <markus.lehto...@linux.intel.com> --- meta/recipes-devtools/images/python-pgo-image.bb | 57 +++- 1 file changed, 45 insertions(+), 12 deletions(-) diff --git a/meta/recipes-devtools/images/python-pgo-image

[OE-core] [PATCH v2 05/20] devtools/images: add python-pgo-image

2017-05-16 Thread Markus Lehtonen
by specifying ${PYTHON_PROFILE_TASK}. [YOCTO #9338] Signed-off-by: Markus Lehtonen <markus.lehto...@linux.intel.com> --- meta/recipes-devtools/images/python-pgo-image.bb | 62 1 file changed, 62 insertions(+) create mode 100644 meta/recipes-devtools/images/python-pgo-im

[OE-core] [PATCH v2 00/20] support profile-optimized build for Python

2017-05-16 Thread Markus Lehtonen
-core-contrib marquiz/fixes-9338 http://cgit.openembedded.org/openembedded-core-contrib/log/?h=marquiz/fixes-9338 Markus Lehtonen (20): python-native: support profile optimized build python: add python-profile-opt recipe python: remove path hack from setup.py python-profile-opt: rename

[OE-core] [PATCH v2 01/20] python-native: support profile optimized build

2017-05-16 Thread Markus Lehtonen
OFILE_TASK = "${S}/Tools/pybench/pybench.py -n 2 --with-gc --with-syscheck" [YOCTO #9338] Signed-off-by: Markus Lehtonen <markus.lehto...@linux.intel.com> --- ...x-.so-loading-when-when-running-profile-t.patch | 26 ++ .../python/python-native_2.7.13.bb

[OE-core] [PATCH v2 03/20] python: remove path hack from setup.py

2017-05-16 Thread Markus Lehtonen
and headers. [YOCTO #9338] Signed-off-by: Markus Lehtonen <markus.lehto...@linux.intel.com> --- .../python/python/01-use-proper-tools-for-cross-build.patch| 10 -- 1 file changed, 10 deletions(-) diff --git a/meta/recipes-devtools/python/python/01-use-proper-tools-for-cross-build.pa

[OE-core] [PATCH v2 06/20] python: make profile-optimized build possible

2017-05-16 Thread Markus Lehtonen
PYTHON_PROFILE_OPT = "1" in your local.conf. Be sure to get the profile data for python, preferably by running "bitbake python-pgo-image -c profile". [YOCTO #9338] Signed-off-by: Markus Lehtonen <markus.lehto...@linux.intel.com> --- meta/recipes-devtools/py

[OE-core] [PATCH v2 07/20] python-pgo-image: exclude tests from the default profile target

2017-05-16 Thread Markus Lehtonen
] Signed-off-by: Markus Lehtonen <markus.lehto...@linux.intel.com> --- meta/recipes-devtools/images/python-pgo-image.bb | 9 - 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/meta/recipes-devtools/images/python-pgo-image.bb b/meta/recipes-devtools/images/python-pgo-image.bb

[OE-core] [PATCH v2 02/20] python: add python-profile-opt recipe

2017-05-16 Thread Markus Lehtonen
. This patch enables the first step, i.e. building python with profile instrumentation enabled [YOCTO #9338] Signed-off-by: Markus Lehtonen <markus.lehto...@linux.intel.com> --- .../python/python-profile-opt_2.7.13.bb| 17 ++ meta/recipes-devtools/python/

[OE-core] [PATCH v2 04/20] python-profile-opt: rename libpython

2017-05-16 Thread Markus Lehtonen
The python library needs to be renamed so that the automatic dependency generation/checking mechanism in bitbake does not get confused. Otherwise python-profile-opt will depend on libpython from the "normal" python package. [YOCTO #9338] Signed-off-by: Markus Lehtonen <

[OE-core] [PATCH] scripts/create-pull-request: fix handling of '-t' option

2017-05-15 Thread Markus Lehtonen
Signed-off-by: Markus Lehtonen <markus.lehto...@linux.intel.com> --- scripts/create-pull-request | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/create-pull-request b/scripts/create-pull-request index 46d65386a3..5bdea44482 100755 --- a/scripts/create-pull-r

[OE-core] [PATCH 5/5] scripts/buildstats-diff: support optimized rusage values

2017-05-15 Thread Markus Lehtonen
Buildstats from oe-build-perf-test results have been optimized to not have child rusage values at all. There, rusage is the sum of parent and child rusage values. This patch makes buildstats-diff compatible with this format. [YOCTO #11355] Signed-off-by: Markus Lehtonen <markus.le

[OE-core] [PATCH 4/5] oe-build-perf-report: implement --dump-buildstats

2017-05-15 Thread Markus Lehtonen
For dumping buildstats from the test runs being reported. The output directory where buildstats are copied is 'oe-build-perf-buildstats/'. Buildstats can be then further analyzed with buildstats-diff script, for example. [YOCTO #11355] Signed-off-by: Markus Lehtonen <markus.le

[OE-core] [PATCH 3/5] oe-build-perf-report: two verbosity levels for --list

2017-05-15 Thread Markus Lehtonen
Signed-off-by: Markus Lehtonen <markus.lehto...@linux.intel.com> --- scripts/oe-build-perf-report | 45 1 file changed, 33 insertions(+), 12 deletions(-) diff --git a/scripts/oe-build-perf-report b/scripts/oe-build-perf-report index fc32

[OE-core] [PATCH 0/5] Enhancements to build perf reporting

2017-05-15 Thread Markus Lehtonen
: python*-git: Upgrade to version 2.1.3 (2017-05-12 08:49:30 +0100) are available in the git repository at: git://git.openembedded.org/openembedded-core-contrib marquiz/fixes-11355 http://cgit.openembedded.org/openembedded-core-contrib/log/?h=marquiz/fixes-11355 Markus Lehtonen (5

[OE-core] [PATCH 2/5] oe-build-perf-report: use correct x-axis max value in html charts

2017-05-15 Thread Markus Lehtonen
Signed-off-by: Markus Lehtonen <markus.lehto...@linux.intel.com> --- scripts/oe-build-perf-report | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/oe-build-perf-report b/scripts/oe-build-perf-report index ced5ff2e12..fc32fbe39a 100755 --- a/scripts/oe-build-perf-

[OE-core] [PATCH 1/5] oeqa.utils.git: use --verify in rev_parse()

2017-05-15 Thread Markus Lehtonen
We use rev-parse for turning git object names into SHA-1 and checking their existence. Using --verify option makes sure git-rev-parse does what we expect. Signed-off-by: Markus Lehtonen <markus.lehto...@linux.intel.com> --- meta/lib/oeqa/utils/git.py | 2 +- 1 file changed, 1 insertion

[OE-core] [PATCH 2/2] oe-build-perf-report: allow slashes in {branch} field in tag names

2017-05-04 Thread Markus Lehtonen
The target branch name might contain slashes. Signed-off-by: Markus Lehtonen <markus.lehto...@linux.intel.com> --- scripts/oe-build-perf-report | 1 + 1 file changed, 1 insertion(+) diff --git a/scripts/oe-build-perf-report b/scripts/oe-build-perf-report index 876e3b1a76..d460caae18

[OE-core] [PATCH 1/2] oe-build-perf-report: accept parenthesis in tag names

2017-05-04 Thread Markus Lehtonen
Signed-off-by: Markus Lehtonen <markus.lehto...@linux.intel.com> --- scripts/oe-build-perf-report | 6 -- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/scripts/oe-build-perf-report b/scripts/oe-build-perf-report index ced5ff2e12..876e3b1a76 100755 --- a/scripts/oe-buil

[OE-core] [PATCH] build-perf-test-wrapper.sh: support uploading test reports

2017-05-04 Thread Markus Lehtonen
Implement new '-R' command line option for specifying an rsync destination where to upload test reports. [YOCTO #5049] Signed-off-by: Markus Lehtonen <markus.lehto...@linux.intel.com> --- scripts/contrib/build-perf-test-wrapper.sh | 30 +- 1 file chang

[OE-core] [PATCH v2 1/1] oe.lsb: add get_os_release()

2017-04-27 Thread Markus Lehtonen
Move get_os_release() from oeqa.utils.metadata to oe.lsb, merging the code with release_dict_osr() from oe.lsb. This removes some code duplication and makes get_os_release() more robust. Signed-off-by: Markus Lehtonen <markus.lehto...@linux.intel.com> --- meta/lib/oe/lsb.py

[OE-core] [PATCH v2 0/1] oeqa.utils.metadata: cope with invalid lines in os-release file

2017-04-27 Thread Markus Lehtonen
Total rewrite of the patch, as per Joshua's comments. Refactor and move get_os_release() from oeqa.utils.metadata to oe.lsb. Markus Lehtonen (1): oe.lsb: add get_os_release() meta/lib/oe/lsb.py | 33 - meta/lib/oeqa/utils/metadata.py | 16

Re: [OE-core] [PATCH] oeqa.utils.metadata: cope with invalid lines in os-release file

2017-04-27 Thread Markus Lehtonen
On 26/04/2017, 17.47, "Joshua Lock" <joshua.g.l...@linux.intel.com> wrote: On Wed, 2017-04-26 at 17:39 +0300, Markus Lehtonen wrote: > Don't crash if every line in /etc/os-release does not adhere to the > expected "key=val" format. E.g. Cent

[OE-core] [PATCH] oeqa.utils.metadata: cope with invalid lines in os-release file

2017-04-26 Thread Markus Lehtonen
Don't crash if every line in /etc/os-release does not adhere to the expected "key=val" format. E.g. CentOS 7 has empty lines in the file. Signed-off-by: Markus Lehtonen <markus.lehto...@linux.intel.com> --- meta/lib/oeqa/utils/metadata.py | 6 -- 1 file changed, 4 insertions

[OE-core] [PATCH] oe-build-perf-report-email.py: use pwd for getting user name

2017-04-12 Thread Markus Lehtonen
executed from a cron script, for example. Signed-off-by: Markus Lehtonen <markus.lehto...@linux.intel.com> --- scripts/contrib/oe-build-perf-report-email.py | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/scripts/contrib/oe-build-perf-report-email.py b/scripts/contrib/oe

[OE-core] [PATCH 3/3] scripts/oe-build-perf-report: improve guessing of args

2017-04-07 Thread Markus Lehtonen
Search remote branches, too, when finding the latest commit. Signed-off-by: Markus Lehtonen <markus.lehto...@linux.intel.com> --- scripts/oe-build-perf-report | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/oe-build-perf-report b/scripts/oe-build-perf-report

[OE-core] [PATCH 0/3] Fixes to build perf test reporting

2017-04-07 Thread Markus Lehtonen
Three miscellaneous one-liners fixing minor issues in build performance test reporting scripts. Markus Lehtonen (3): build-perf-test-wrapper.sh: support extra args for email script oe-build-perf-report-email.py: use proper fallback email address scripts/oe-build-perf-report: improve

[OE-core] [PATCH 2/3] oe-build-perf-report-email.py: use proper fallback email address

2017-04-07 Thread Markus Lehtonen
Use properly formatted fallback email address instead of just the username. Signed-off-by: Markus Lehtonen <markus.lehto...@linux.intel.com> --- scripts/contrib/oe-build-perf-report-email.py | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/scripts/contrib/oe-buil

[OE-core] [PATCH 1/3] build-perf-test-wrapper.sh: support extra args for email script

2017-04-07 Thread Markus Lehtonen
Make it possible to provide (extra) command line arguments to the oe-build-perf-test-email script via a new environment variable OE_BUILD_PERF_REPORT_EMAIL_EXTRA_ARGS. Signed-off-by: Markus Lehtonen <markus.lehto...@linux.intel.com> --- scripts/contrib/build-perf-test-wrapper.sh | 2 +-

[OE-core] [PATCH] oe-build-perf-report-email.py: fix one file path

2017-04-06 Thread Markus Lehtonen
Sending report email was not working correctly if the script was given an html report path that contained directory components. Signed-off-by: Markus Lehtonen <markus.lehto...@linux.intel.com> --- scripts/contrib/oe-build-perf-report-email.py | 2 +- 1 file changed, 1 insertion(+), 1 de

[OE-core] [PATCH v2 0/8] Build perf test enhancements

2017-04-04 Thread Markus Lehtonen
://git.openembedded.org/openembedded-core-contrib marquiz/buildperf/scripts http://git.openembedded.org/openembedded-core-contrib/log/?h=marquiz/buildperf/scripts Markus Lehtonen (8): scripts/oe-build-perf-report: better guessing of args scripts/oe-build-perf-report: fix 'charts ready

[OE-core] [PATCH 7/7] build-perf-test-wrapper.sh: support sending email reports

2017-04-03 Thread Markus Lehtonen
Add new '-E' command line option for sending an email report to specified recipient. Signed-off-by: Markus Lehtonen <markus.lehto...@linux.intel.com> --- scripts/contrib/build-perf-test-wrapper.sh | 20 +++- 1 file changed, 19 insertions(+), 1 deletion(-) diff --git a/s

[OE-core] [PATCH 6/7] build-perf-test-wrapper.sh: support pushing to remote Git

2017-04-03 Thread Markus Lehtonen
Implement new '-P' option for spefifying a Git remote where to push results after committing to a local Git repository. Signed-off-by: Markus Lehtonen <markus.lehto...@linux.intel.com> --- scripts/contrib/build-perf-test-wrapper.sh | 7 ++- 1 file changed, 6 insertions(+), 1 de

[OE-core] [PATCH 5/7] build-perf-test-wrapper.sh: correctly check test script exit status

2017-04-03 Thread Markus Lehtonen
Test of the exit code was accidentally moved to wrong place when oe-git-archive was taken into use. Signed-off-by: Markus Lehtonen <markus.lehto...@linux.intel.com> --- scripts/contrib/build-perf-test-wrapper.sh | 16 1 file changed, 8 insertions(+), 8 deletions(-) diff

[OE-core] [PATCH 4/7] build-perf-test-wrapper.sh: make it possible to specify Git branch name

2017-04-03 Thread Markus Lehtonen
Support : format for the -c argument. This makes it possible to test older commits of a certain branch (not just the tip of it) so that the branch name will still be correctly recorded in the test report data. Signed-off-by: Markus Lehtonen <markus.lehto...@linux.intel.com> --- scripts/c

[OE-core] [PATCH 2/7] scripts/oe-build-perf-report: fix 'charts ready' console message

2017-04-03 Thread Markus Lehtonen
, in turn, caused oe-build-perf-report-email.py script to fail with a timeout. Signed-off-by: Markus Lehtonen <markus.lehto...@linux.intel.com> --- scripts/lib/build_perf/html/measurement_chart.html | 6 +++--- scripts/lib/build_perf/html/report.html| 5 + 2 files changed, 4 inse

[OE-core] [PATCH 3/7] scripts/oe-build-perf-report: fix typo

2017-04-03 Thread Markus Lehtonen
Signed-off-by: Markus Lehtonen <markus.lehto...@linux.intel.com> --- scripts/oe-build-perf-report | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/oe-build-perf-report b/scripts/oe-build-perf-report index ca9cf1d..6f2db49 100755 --- a/scripts/oe-build-perf-report

[OE-core] [PATCH 1/7] scripts/oe-build-perf-report: better guessing of args

2017-04-03 Thread Markus Lehtonen
When getting info from the latest commit, don't search all refs but only branches. We don't get correct data from refs/tags/* or refs/notest/*, for example. Signed-off-by: Markus Lehtonen <markus.lehto...@linux.intel.com> --- scripts/oe-build-perf-report | 2 +- 1 file changed, 1 insertion

[OE-core] [PATCH 0/7] Build perf test enhancements

2017-04-03 Thread Markus Lehtonen
://git.openembedded.org/openembedded-core-contrib marquiz/buildperf/scripts http://git.openembedded.org/openembedded-core-contrib/log/?h=marquiz/buildperf/scripts Markus Lehtonen (7): scripts/oe-build-perf-report: better guessing of args scripts/oe-build-perf-report: fix 'charts ready' console message

[OE-core] [PATCH 1/2] scripts: add oe-build-perf-report script

2017-03-31 Thread Markus Lehtonen
for generating HTML reports so it requires python3-jinja2 to be installed on the system. [YOCTO #10931] Signed-off-by: Markus Lehtonen <markus.lehto...@linux.intel.com> --- scripts/lib/build_perf/__init__.py | 31 ++ scripts/lib/build_perf/html.py | 19 + s

[OE-core] [PATCH 0/2] Tools for visualizing build performance test results

2017-03-31 Thread Markus Lehtonen
at: git://git.openembedded.org/openembedded-core-contrib marquiz/buildperf/scripts http://git.openembedded.org/openembedded-core-contrib/log/?h=marquiz/buildperf/scripts Markus Lehtonen (2): scripts: add oe-build-perf-report script scripts/contrib: add oe-build-perf-report-email scripts

[OE-core] [PATCH 2/2] scripts/contrib: add oe-build-perf-report-email

2017-03-31 Thread Markus Lehtonen
: - phantomjs - optipng [YOCTO #10931] Signed-off-by: Markus Lehtonen <markus.lehto...@linux.intel.com> --- scripts/contrib/oe-build-perf-report-email.py | 266 ++ scripts/lib/build_perf/scrape-html-report.js | 56 ++ 2 files changed, 322 insertions(+) creat

[OE-core] [PATCH 2/2] oeqa.buildperf: limit the length of error output

2017-03-24 Thread Markus Lehtonen
Limit the length of error logs to 40 lines. We don't need to show/archive thousands of lines of bitbake logs if an error occurs. Signed-off-by: Markus Lehtonen <markus.lehto...@linux.intel.com> --- meta/lib/oeqa/buildperf/base.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff

[OE-core] [PATCH 1/2] oeqa.utils.commands: limit runCmd exception output

2017-03-24 Thread Markus Lehtonen
Make it possible to limit the length of output lines shown in runCmd exceptions. E.g when running bitbake we easily get thousands of lines of log output, where only the last few (tens) are interesting or relevant when an error occurs. Signed-off-by: Markus Lehtonen <markus.le

[OE-core] [PATCH 0/6] oe-build-perf-test: re-organize buildstats archiving

2017-03-24 Thread Markus Lehtonen
bedded.org/openembedded-core-contrib marquiz/buildperf/buildstats http://git.openembedded.org/openembedded-core-contrib/log/?h=marquiz/buildperf/buildstats Markus Lehtonen (6): oe-build-perf-test: pack all buildstat in one file oe-build-perf-test: sum rusage in buildstats scripts/oe-git-ar

[OE-core] [PATCH 1/6] oe-build-perf-test: pack all buildstat in one file

2017-03-24 Thread Markus Lehtonen
re no more per-testcase data files to store permanently. [YOCTO #10582] Signed-off-by: Markus Lehtonen <markus.lehto...@linux.intel.com> --- meta/lib/oeqa/buildperf/base.py | 42 --- meta/lib/oeqa/buildperf/test_basic.py | 24 +--- scrip

[OE-core] [PATCH 5/6] scripts/oe-git-archive: implement --notes

2017-03-24 Thread Markus Lehtonen
Option for adding git-notes to the commit. [YOCTO #10582] Signed-off-by: Markus Lehtonen <markus.lehto...@linux.intel.com> --- scripts/oe-git-archive | 21 ++--- 1 file changed, 18 insertions(+), 3 deletions(-) diff --git a/scripts/oe-git-archive b/scripts/oe-git-archive

[OE-core] [PATCH 6/6] build-perf-test-wrapper.sh: store buildstats under git-notes

2017-03-24 Thread Markus Lehtonen
In order to dramatically reduce the data footprint of the result branches. This makes cloning/pulling result repositories a lot faster. Buildstats can still be accessed, when needed by doing git fetch refs/notes/buildstats/*:refs/notes/buildstats/* [YOCTO #10582] Signed-off-by: Markus Lehtonen

[OE-core] [PATCH 2/6] oe-build-perf-test: sum rusage in buildstats

2017-03-24 Thread Markus Lehtonen
Instead of separate rusage and child rusage values, only store their sum value in buildstats. This is a big reduction in data footprint without really losing any interesting data. Also, utilize OrderedDict to order data more logically. [YOCTO #10582] Signed-off-by: Markus Lehtonen <markus.le

[OE-core] [PATCH 4/6] scripts/oe-git-archive: implement --exclude

2017-03-24 Thread Markus Lehtonen
May be used for excluding certain files from the commit. [YOCTO #10582] Signed-off-by: Markus Lehtonen <markus.lehto...@linux.intel.com> --- scripts/oe-git-archive | 12 ++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/scripts/oe-git-archive b/scripts/oe-git-a

[OE-core] [PATCH 3/6] scripts/oe-git-archive: fix pushing

2017-03-24 Thread Markus Lehtonen
Git arguments were badly laid out. Signed-off-by: Markus Lehtonen <markus.lehto...@linux.intel.com> --- scripts/oe-git-archive | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/scripts/oe-git-archive b/scripts/oe-git-archive index 117118b..aef4ba1 100755 --- a/scri

[OE-core] [morty][PATCH 3/4] lib/oe/gpg_sign: make gpg version a property of the signer

2017-03-03 Thread Markus Lehtonen
Signed-off-by: Markus Lehtonen <markus.lehto...@linux.intel.com> Signed-off-by: Ross Burton <ross.bur...@intel.com> (cherry picked from commit a00a362e3dc18ba04230cbbd6f91264e5d76f40d) --- meta/lib/oe/gpg_sign.py | 10 +- 1 file changed, 5 insertions(+), 5 deletions(-) diff

[OE-core] [morty][PATCH 1/4] lib/oe/gpg_sign: sign rpm packages in chunks of 100

2017-03-03 Thread Markus Lehtonen
Split the file list into chunks in order to avoid "OSError: [Errno 7] Argument list too long" This would happend when a package has huge amount of subpackages, e.g. glibc-locale. [YOCTO #11069] Signed-off-by: Markus Lehtonen <markus.lehto...@linux.intel.com> Signed-off-by: Ross

[OE-core] [morty][PATCH 0/4] rpm signing fixes

2017-03-03 Thread Markus Lehtonen
rty Markus Lehtonen (4): lib/oe/gpg_sign: sign rpm packages in chunks of 100 rpm: support customizing gpg command line lib/oe/gpg_sign: make gpg version a property of the signer lib/oe/gpg_sign: fix rpm signing with gpg > 2.1 meta/lib/oe/gpg_sign.py

[OE-core] [morty][PATCH 4/4] lib/oe/gpg_sign: fix rpm signing with gpg > 2.1

2017-03-03 Thread Markus Lehtonen
We need to check the gpg version and alter its command line options accordingly. [YOCTO #11054] Signed-off-by: Markus Lehtonen <markus.lehto...@linux.intel.com> Signed-off-by: Ross Burton <ross.bur...@intel.com> (cherry picked from commit 44a44b7e582a5a654baf21829d168568481c13d9) -

[OE-core] [morty][PATCH 2/4] rpm: support customizing gpg command line

2017-03-03 Thread Markus Lehtonen
Add a new %_gpg_sign_cmd_extra_args macro that allows customizing the gpg options used when signing rpm packages. This is needed to be able to sign packages with gpg 2.1 which requires "--pinentry-mode loopback" to allow non-interactive signing. [YOCTO #11054] Signed-off-by: Marku

Re: [OE-core] [PATCH 0/3] Fix rpm signing with GPG v2.1

2017-02-23 Thread Markus Lehtonen
On 23/02/2017, 17.58, "akuster808" <akuster...@gmail.com> wrote: Markus, On 02/23/2017 02:24 AM, Markus Lehtonen wrote: > This patchset makes signing work with all versions of GPG. Previously, rpm > package signing in oe-core was not workin

[OE-core] [PATCH] build-perf-test-wrapper.sh: support xml report format

2017-02-23 Thread Markus Lehtonen
Add new command line option '-x' that enbles xml-formatted reports. Signed-off-by: Markus Lehtonen <markus.lehto...@linux.intel.com> --- scripts/contrib/build-perf-test-wrapper.sh | 7 ++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/scripts/contrib/build-perf-test-wrap

[OE-core] [PATCH 3/3] lib/oe/gpg_sign: fix rpm signing with gpg > 2.1

2017-02-23 Thread Markus Lehtonen
We need to check the gpg version and alter its command line options accordingly. [YOCTO #11054] Signed-off-by: Markus Lehtonen <markus.lehto...@linux.intel.com> --- meta/lib/oe/gpg_sign.py | 2 ++ 1 file changed, 2 insertions(+) diff --git a/meta/lib/oe/gpg_sign.py b/meta/lib/oe/gpg_s

[OE-core] [PATCH 2/3] lib/oe/gpg_sign: make gpg version a property of the signer

2017-02-23 Thread Markus Lehtonen
Signed-off-by: Markus Lehtonen <markus.lehto...@linux.intel.com> --- meta/lib/oe/gpg_sign.py | 10 +- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/meta/lib/oe/gpg_sign.py b/meta/lib/oe/gpg_sign.py index dcd1990..21dc5ea 100644 --- a/meta/lib/oe/gpg_sign.py +++ b/me

[OE-core] [PATCH 1/3] rpm: support customizing gpg command line

2017-02-23 Thread Markus Lehtonen
Add a new %_gpg_sign_cmd_extra_args macro that allows customizing the gpg options used when signing rpm packages. This is needed to be able to sign packages with gpg 2.1 which requires "--pinentry-mode loopback" to allow non-interactive signing. [YOCTO #11054] Signed-off-by: Marku

[OE-core] [PATCH] license.bbclass: run chown only under pseudo

2017-02-22 Thread Markus Lehtonen
running under pseudo. [YOCTO #10897] Signed-off-by: Markus Lehtonen <markus.lehto...@linux.intel.com> --- meta/classes/license.bbclass | 15 +++ 1 file changed, 3 insertions(+), 12 deletions(-) diff --git a/meta/classes/license.bbclass b/meta/classes/license.bbclass index 9

[OE-core] [PATCH 17/20] python3: fix profile-optimized build of modules

2017-02-20 Thread Markus Lehtonen
Without this the pgo-related compiler flags are not used in cross-builds. [YOCTO #9338] Signed-off-by: Markus Lehtonen <markus.lehto...@linux.intel.com> --- ...-CFLAGS-for-extensions-when-cross-compili.patch | 56 ++ meta/recipes-devtools/python/python3_3.5.2.bb

[OE-core] [PATCH 15/20] python3: remove two setup.py cross-compile hacks

2017-02-20 Thread Markus Lehtonen
en running "gcc -E -v". The second hack becomes useless after the first hack has been removed and we get the standard include and lib directories normally from gcc. [YOCTO #9338] Signed-off-by: Markus Lehtonen <markus.lehto...@linux.intel.com> --- .../python3-setup.py-no-host-

[OE-core] [PATCH 20/20] python-pgo-image: change python3 profile target to pybench

2017-02-20 Thread Markus Lehtonen
Running test.regrtest was still taking ridiculously long, i.e. around 4 hours in qemux86 on my desktop machine. Change to pybench which "only" takes around 55 minutes. [YOCTO #9338] Signed-off-by: Markus Lehtonen <markus.lehto...@linux.intel.com> --- meta/recipes-devtools/i

[OE-core] [PATCH 19/20] python3: add python3-tools subpackage

2017-02-20 Thread Markus Lehtonen
[YOCTO #9338] Signed-off-by: Markus Lehtonen <markus.lehto...@linux.intel.com> --- meta/recipes-devtools/python/python-3.5-manifest.inc | 8 +--- meta/recipes-devtools/python/python3_3.5.2.bb| 5 + 2 files changed, 10 insertions(+), 3 deletions(-) diff --git a/meta/r

[OE-core] [PATCH 16/20] python3: support profile optimized build

2017-02-20 Thread Markus Lehtonen
e data can be obtainen e.g. by running bitbake python-pgo-image -c profile3. [YOCTO #9338] Signed-off-by: Markus Lehtonen <markus.lehto...@linux.intel.com> --- meta/recipes-devtools/python/python3_3.5.2.bb | 20 1 file changed, 20 insertions(+) diff --git a/meta/recipes

[OE-core] [PATCH 18/20] python-pgo-image: exclude tests from the python3 profile target

2017-02-20 Thread Markus Lehtonen
test_urllib2_localnet: 1276.8s test_itertools: 944.7s test_tuple: 847.7s test_trace: 756.7s test_tarfile: 716.7s test_unicodedata: 709.9s test_decimal: 615.1s [YOCTO #9338] Signed-off-by: Markus Lehtonen <markus.lehto...@linux.intel.com> --- meta/recipes-devtools/images/pyth

[OE-core] [PATCH 14/20] python-pgo-image: profiling for python3

2017-02-20 Thread Markus Lehtonen
with PYTHON3_PROFILE_TASK. [YOCTO #9338] Signed-off-by: Markus Lehtonen <markus.lehto...@linux.intel.com> --- meta/recipes-devtools/images/python-pgo-image.bb | 55 +++- 1 file changed, 44 insertions(+), 11 deletions(-) diff --git a/meta/recipes-devtools/images/python-pgo-image

<    1   2   3   4   5   6   7   >