Re: [OE-core] [PATCH 1/3] systemtap: upgrade 4.9 -> 5.0

[Victor Kamensky]

On Thu, Dec 14, 2023 at 6:27 AM Martin Jansa  wrote:
>
> PACKAGECONFIG:remove:pn-elfutils = "debuginfod"
> PACKAGECONFIG:remove:pn-elfutils = "libdebuginfod"
> PACKAGECONFIG:remove:pn-binutils = "debuginfod"
>
> in local.conf is the easiest way to reproduce it.
>
> The one for binutils isn't strictly needed (but target binutils will fail as 
> well when debuginfod is disabled in elfutils and enabled in binutils).

Thanks, Martin. I was able to reproduce it with removing debuginfod
from DISTRO_FEATURES_DEFAULT

Submitted patches to systemtap mailing list:
https://sourceware.org/pipermail/systemtap/2023q4/027914.html
https://sourceware.org/pipermail/systemtap/2023q4/027915.html

And to oe-core:
https://lists.openembedded.org/g/openembedded-core/message/192600?p=%2C%2C%2C20%2C0%2C0%2C0%3A%3Acreated%2C0%2CVictor+Kamensky%2C20%2C2%2C0%2C103238308
https://lists.openembedded.org/g/openembedded-core/message/192601?p=%2C%2C%2C20%2C0%2C0%2C0%3A%3Acreated%2C0%2CVictor+Kamensky%2C20%2C2%2C0%2C103238309

Appreciate reporting the issue!

Thanks,
Victor

> On Wed, Dec 13, 2023 at 11:45 PM Martin Jansa via lists.openembedded.org 
>  wrote:
>>
>> Hi Victor,
>>
>> no problem, at least it forced me to enable debuginfod in our builds as well 
>> (where it wasn't intentionally disabled). Just didn't get enabled when it 
>> was added to default DISTRO_FEATUREs as we define our own and it wasn't 
>> using BACKFILL mechanism to enable it in such setups.
>>
>> Removing debuginfod from DISTRO_FEATURES should be enough to reproduce the 
>> issue (as it will disable both *debuginfod PACKAGECONFIG options from 
>> elfutils as well).
>>
>> I haven't tried, but removing the PACKAGECONFIG just from elfutils will 
>> probably reproduce it as well (with possibly fewer recipes needing to 
>> rebuilt). There are 2 PACKAGECONFIGs for this, not sure which one is needed, 
>> but you can just disable both.
>>
>> Cheers,
>>
>> On Wed, Dec 13, 2023 at 10:27 PM Victor Kamensky 
>>  wrote:
>>>
>>> Hi Martin,
>>>
>>> Sorry about that. I am looking at it now.
>>>
>>> Do I understand it correctly to reproduce the issue: I need to remove
>>> debuginfod (as it is currently on
>>> by default) from DISTO_FEATURES and remove debuginfod from elfutils
>>> PACKAGECONFIG defaults?
>>>
>>> Thanks,
>>> Victor
>>>
>>> On Sun, Dec 10, 2023 at 3:16 AM Martin Jansa  wrote:
>>> >
>>> > This seems to be causing:
>>> > ../git/util.cxx:51:10: fatal error: elfutils/debuginfod.h: No such file 
>>> > or directory
>>> >51 | #include 
>>> >   |  ^~~
>>> > compilation terminated.
>>> > make[2]: *** [Makefile:1389: stap-util.o] Error 1
>>> >
>>> > on some systems.
>>> >
>>> > Looks like with 
>>> > https://sourceware.org/git/?p=systemtap.git;a=commit;h=2e67b053e3796ee7cf29a39f9698729b52078406
>>> >  it now enables debuginfod ("AC_MSG_RESULT([yes])" _after_ the 
>>> > PKG_CHECK_MODULES now seems wrong).
>>> >
>>> > log.do_configure:
>>> > checking whether to use debuginfod... checking for debuginfod... no
>>> > yes
>>> > checking for debuginfod_find_metadata in -ldebuginfod... no
>>> >
>>> > build $ grep DEBUGINFO config.h
>>> > #define HAVE_LIBDEBUGINFOD 1
>>> >
>>> > config.log (see the strange result yes):
>>> > configure:12213: checking whether to use debuginfod
>>> > configure:12228: checking for debuginfod
>>> > configure:12235: $PKG_CONFIG --exists --print-errors "libdebuginfod >= 
>>> > 0.179"
>>> > Package libdebuginfod was not found in the pkg-config search path.
>>> > Perhaps you should add the directory containing `libdebuginfod.pc'
>>> > to the PKG_CONFIG_PATH environment variable
>>> > No package 'libdebuginfod' found
>>> > configure:12238: $? = 1
>>> > configure:12252: $PKG_CONFIG --exists --print-errors "libdebuginfod >= 
>>> > 0.179"
>>> > Package libdebuginfod was not found in the pkg-config search path.
>>> > Perhaps you should add the directory containing `libdebuginfod.pc'
>>> > to the PKG_CONFIG_PATH environment variable
>>> > No package 'libdebuginfod' found
>>> > configure:12255: $? = 1
>>> > configure:12269: result: no
>>> > No package 'libdebuginfod' found
>

[OE-core] [PATCH 2/2] systemtap: fix libdebuginfod auto detection logic

[Victor Kamensky]

Addresses an issue when systemtap is built in situation where debuginfod
is not part of DISTRO_FEATURES and latest SystemTap has problem of
auto detecting libdebuginfod library as it was reported by Martin
Jansa in [1].

[1] 
https://lists.openembedded.org/g/openembedded-core/message/192109?p=%2C%2C%2C20%2C0%2C0%2C0%3A%3Acreated%2C0%2Csystemtap%2C20%2C2%2C0%2C102987514

Signed-off-by: Victor Kamensky 
---
 ...-broken-libdebuginfod-library-auto-d.patch | 51 +++
 .../systemtap/systemtap_git.inc   |  1 +
 2 files changed, 52 insertions(+)
 create mode 100644 
meta/recipes-kernel/systemtap/systemtap/0001-configure.ac-fix-broken-libdebuginfod-library-auto-d.patch

diff --git 
a/meta/recipes-kernel/systemtap/systemtap/0001-configure.ac-fix-broken-libdebuginfod-library-auto-d.patch
 
b/meta/recipes-kernel/systemtap/systemtap/0001-configure.ac-fix-broken-libdebuginfod-library-auto-d.patch
new file mode 100644
index 00..98641826f6
--- /dev/null
+++ 
b/meta/recipes-kernel/systemtap/systemtap/0001-configure.ac-fix-broken-libdebuginfod-library-auto-d.patch
@@ -0,0 +1,51 @@
+From 3913ad3e28a19811e1b52338112344a487057e4f Mon Sep 17 00:00:00 2001
+From: Victor Kamensky 
+Date: Mon, 18 Dec 2023 03:13:38 +
+Subject: [PATCH 1/2] configure.ac: fix broken libdebuginfod library auto
+ detection
+
+After 2e67b053e3796ee7cf29a39f9698729b52078406 "configury: rework debuginfod 
searches"
+commit, libdebuginfod.so library auto detection is broken. It was reported by 
Martin Jansa
+on openembedded-core mailing list [1].
+
+Currently configure.ac does "AC_DEFINE([HAVE_LIBDEBUGINFOD], [1] ..." as long 
as
+no --without-debuginfod option is passed, regardless PKG_CHECK_MODULES check 
result.
+It seems to be bad copy/paste. Address the issue by moving the AC_DEFINE back 
to
+PKG_CHECK_MODULES action-if-found block.
+
+To reproduce the issue on FC system, one can do the following
+"sudo dnf remove elfutils-debuginfod-client-devel" and then try to build 
SystemTap
+util.cxx will fail to compile because of missing elfutils/debuginfod.h because
+config.h will have "#define HAVE_LIBDEBUGINFOD 1", while config.log and 
configure
+output indicates that check for libdebuginfod library failed.
+
+[1] 
https://lists.openembedded.org/g/openembedded-core/message/192109?p=%2C%2C%2C20%2C0%2C0%2C0%3A%3Acreated%2C0%2Csystemtap%2C20%2C2%2C0%2C102987514
+
+Upstream-Status: Submitted 
[https://sourceware.org/pipermail/systemtap/2023q4/027914.html]
+Signed-off-by: Victor Kamensky 
+---
+ configure.ac | 5 ++---
+ 1 file changed, 2 insertions(+), 3 deletions(-)
+
+diff --git a/configure.ac b/configure.ac
+index d9559c5c3..18cd7f84a 100644
+--- a/configure.ac
 b/configure.ac
+@@ -219,12 +219,11 @@ dnl take the user at his or her word
+ elif test "x$with_debuginfod" != xno; then
+ dnl check in the system pkgconfig
+   PKG_CHECK_MODULES([debuginfod], [libdebuginfod >= 0.179],
+-[have_debuginfod=1],
++[have_debuginfod=1
++ AC_DEFINE([HAVE_LIBDEBUGINFOD], [1], [Define to 1 if debuginfod is 
enabled.])],
+ [if test "x$with_debuginfod" = xyes; then
+AC_MSG_ERROR(["--with-debuginfod was given, but libdebuginfod is 
missing or unusable."])
+  fi])
+-  AC_DEFINE([HAVE_LIBDEBUGINFOD], [1], [Define to 1 if debuginfod is 
enabled.])
+-  AC_MSG_RESULT([yes])  
+ else
+   AC_MSG_RESULT([no])
+ fi
+-- 
+2.31.1
+
diff --git a/meta/recipes-kernel/systemtap/systemtap_git.inc 
b/meta/recipes-kernel/systemtap/systemtap_git.inc
index 9fe85676fc..bebfa7f777 100644
--- a/meta/recipes-kernel/systemtap/systemtap_git.inc
+++ b/meta/recipes-kernel/systemtap/systemtap_git.inc
@@ -9,6 +9,7 @@ SRC_URI = 
"git://sourceware.org/git/systemtap.git;branch=master;protocol=https \

file://0001-staprun-stapbpf-don-t-support-installing-a-non-root.patch \

file://0001-Makefile.am-remove-runtime-linux-uprobes-and-runtime.patch \
file://0001-prerelease-datestamp-fixes.patch \
+   
file://0001-configure.ac-fix-broken-libdebuginfod-library-auto-d.patch \
"
 
 COMPATIBLE_HOST = 
'(x86_64|i.86|powerpc|arm|aarch64|microblazeel|mips|riscv64).*-linux'
-- 
2.31.1


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#192601): 
https://lists.openembedded.org/g/openembedded-core/message/192601
Mute This Topic: https://lists.openembedded.org/mt/103238309/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core] [PATCH 1/2] systemtap: explicit handling debuginfod library dependency

[Victor Kamensky]

After systemtap commit 2e67b053e3796ee7cf29a39f9698729b52078406
"configury: rework debuginfod searches" automatic detection of
libdebuginfod is broken. Let's handle it explicitly with debuginfod
package feature. It is good to have such feature anyway.

Signed-off-by: Victor Kamensky 
---
 meta/recipes-kernel/systemtap/systemtap_git.bb | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/meta/recipes-kernel/systemtap/systemtap_git.bb 
b/meta/recipes-kernel/systemtap/systemtap_git.bb
index d320a8a5e0..0a1349e128 100644
--- a/meta/recipes-kernel/systemtap/systemtap_git.bb
+++ b/meta/recipes-kernel/systemtap/systemtap_git.bb
@@ -24,12 +24,13 @@ STAP_DOCS ?= "--disable-docs --disable-publican 
--disable-refdocs"
 
 EXTRA_OECONF += "${STAP_DOCS} "
 
-PACKAGECONFIG ??= "translator sqlite monitor python3-probes"
+PACKAGECONFIG ??= "translator sqlite monitor python3-probes 
${@bb.utils.filter('DISTRO_FEATURES', 'debuginfod', d)}"
 PACKAGECONFIG[translator] = 
"--enable-translator,--disable-translator,boost,bash"
 PACKAGECONFIG[libvirt] = "--enable-libvirt,--disable-libvirt,libvirt"
 PACKAGECONFIG[sqlite] = "--enable-sqlite,--disable-sqlite,sqlite3"
 PACKAGECONFIG[monitor] = "--enable-monitor,--disable-monitor,ncurses json-c"
 PACKAGECONFIG[python3-probes] = 
"--with-python3-probes,--without-python3-probes,python3-setuptools-native"
+PACKAGECONFIG[debuginfod] = "--with-debuginfod, --without-debuginfod"
 
 inherit autotools gettext pkgconfig systemd
 inherit ${@bb.utils.contains('PACKAGECONFIG', 'python3-probes', 
'setuptools3-base', '', d)}
-- 
2.31.1


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#192600): 
https://lists.openembedded.org/g/openembedded-core/message/192600
Mute This Topic: https://lists.openembedded.org/mt/103238308/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [OE-core] [PATCH 1/3] systemtap: upgrade 4.9 -> 5.0

[Victor Kamensky]

file or directory
> collect2: error: ld returned 1 exit status
>
> With debuginfod in DISTRO_FEATURES (which is included by default) and 
> *debuginfod PACKAGECONFIG enabled in elfutils (also by default based on 
> DISTRO_FEAUTRES) this builds OK.
>
> Regards,
>
>
> On Tue, Dec 5, 2023 at 7:33 AM Victor Kamensky  
> wrote:
>>
>> Similar to 4.9 release, in 5.0 release release-5.0 tag is off any branch,
>> release-5.0a tag e72dc118e563c645d93a1a2d771e8d90e8fec1ae is on the master
>> branch. Use this value for SRCREV and add catch up patch,
>> 0001-prerelease-datestamp-fixes.patch, to bring source up to release-5.0 tag
>> level. Remove 0001-prerelease-datestamp-fixes.patch in next upgrade.
>>
>> Signed-off-by: Victor Kamensky 
>> ---
>>  .../0001-prerelease-datestamp-fixes.patch | 23 
>>  .../systemtap/0001-release-date-fix.patch | 37 ---
>>  .../systemtap/systemtap_git.inc   |  6 +--
>>  3 files changed, 26 insertions(+), 40 deletions(-)
>>  create mode 100644 
>> meta/recipes-kernel/systemtap/systemtap/0001-prerelease-datestamp-fixes.patch
>>  delete mode 100644 
>> meta/recipes-kernel/systemtap/systemtap/0001-release-date-fix.patch
>>
>> diff --git 
>> a/meta/recipes-kernel/systemtap/systemtap/0001-prerelease-datestamp-fixes.patch
>>  
>> b/meta/recipes-kernel/systemtap/systemtap/0001-prerelease-datestamp-fixes.patch
>> new file mode 100644
>> index 00..afdc10a3fe
>> --- /dev/null
>> +++ 
>> b/meta/recipes-kernel/systemtap/systemtap/0001-prerelease-datestamp-fixes.patch
>> @@ -0,0 +1,23 @@
>> +From cfc2c1d53924face11e3fab78ded61c359778eb9 Mon Sep 17 00:00:00 2001
>> +From: "Frank Ch. Eigler" 
>> +Date: Sat, 4 Nov 2023 12:19:59 -0400
>> +Subject: [PATCH] prerelease datestamp fixes
>> +
>> +Upstream-Status: Backport
>> +---
>> + NEWS | 2 +-
>> + 1 file changed, 1 insertion(+), 1 deletion(-)
>> +
>> +diff --git a/NEWS b/NEWS
>> +index c7f31e8b7..f0f9056c4 100644
>> +--- a/NEWS
>>  b/NEWS
>> +@@ -1,4 +1,4 @@
>> +-* What's new in version 5.0, 2023-11-03
>> ++* What's new in version 5.0, 2023-11-04
>> +
>> + - Performance improvements in uprobe registration and module startup.
>> +
>> +--
>> +2.31.1
>> +
>> diff --git 
>> a/meta/recipes-kernel/systemtap/systemtap/0001-release-date-fix.patch 
>> b/meta/recipes-kernel/systemtap/systemtap/0001-release-date-fix.patch
>> deleted file mode 100644
>> index 3dcb369ec3..00
>> --- a/meta/recipes-kernel/systemtap/systemtap/0001-release-date-fix.patch
>> +++ /dev/null
>> @@ -1,37 +0,0 @@
>> -From ebb424eee5599fcc131901c0d82d0bfc0d2f57ab Mon Sep 17 00:00:00 2001
>> -From: "Frank Ch. Eigler" 
>> -Date: Fri, 28 Apr 2023 13:51:27 -0400
>> -Subject: [PATCH] release date fix
>> -
>> -Upstream-Status: Backport
>> 
>> - NEWS   | 2 +-
>> - systemtap.spec | 2 +-
>> - 2 files changed, 2 insertions(+), 2 deletions(-)
>> -
>> -diff --git a/NEWS b/NEWS
>> -index 695bc990c..46859e304 100644
>>  a/NEWS
>> -+++ b/NEWS
>> -@@ -1,4 +1,4 @@
>> --* What's new in version 4.9, 2023-04-27
>> -+* What's new in version 4.9, 2023-04-28
>> -
>> - - ISystemtap: the new interactive systemtap jupyter kernel. This provides
>> -   a simple GUI for writing/running Systemtap scripts in an easy, 
>> incremental
>> -diff --git a/systemtap.spec b/systemtap.spec
>> -index 59814f112..fd8023264 100644
>>  a/systemtap.spec
>> -+++ b/systemtap.spec
>> -@@ -1307,7 +1307,7 @@ exit 0
>> -
>> - # PRERELEASE
>> - %changelog
>> --* Thu Apr 27 2023 Frank Ch. Eigler  - 4.9-1
>> -+* Fri Apr 28 2023 Frank Ch. Eigler  - 4.9-1
>> - - Upstream release, see wiki page below for detailed notes.
>> -   https://sourceware.org/systemtap/wiki/SystemTapReleases
>> -
>> ---
>> -2.40.1
>> -
>> diff --git a/meta/recipes-kernel/systemtap/systemtap_git.inc 
>> b/meta/recipes-kernel/systemtap/systemtap_git.inc
>> index de3750ff04..bc2a9ff434 100644
>> --- a/meta/recipes-kernel/systemtap/systemtap_git.inc
>> +++ b/meta/recipes-kernel/systemtap/systemtap_git.inc
>> @@ -1,13 +1,13 @@
>>  LICENSE = "GPL-2.0-only"
>>  LIC_FILES_CHKSUM = "file://COPYING;md5=b234ee4d69f5fce4486a80fdaf4a4263"
>> -SRCREV = "418f0a45ca4473491385b5c7eef777607bbdb3b7"
>> -PV = "4.9"
>> +SRCREV = "e72dc118e563c645d93a1a2d771e8d90e8fec1ae"
>> +PV = "5.0"
>>
>>  SRC_URI = 
>> "git://sourceware.org/git/systemtap.git;branch=master;protocol=https \
>> 
>> file://0001-Do-not-let-configure-write-a-python-location-into-th.patch \
>> file://0001-Install-python-modules-to-correct-library-dir.patch \
>> 
>> file://0001-staprun-stapbpf-don-t-support-installing-a-non-root.patch \
>> -   file://0001-release-date-fix.patch \
>> +   file://0001-prerelease-datestamp-fixes.patch \
>> "
>>
>>  COMPATIBLE_HOST = 
>> '(x86_64|i.86|powerpc|arm|aarch64|microblazeel|mips|riscv64).*-linux'
>> --
>> 2.31.1
>>
>>
>> 
>>

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#192328): 
https://lists.openembedded.org/g/openembedded-core/message/192328
Mute This Topic: https://lists.openembedded.org/mt/102987514/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core] [PATCH 3/3] systemtap-uprobes: removed as obsolete

[Victor Kamensky]

systemtap-uprobes package was not used for a long time - since kernel
itself provided uprobes support. Now source code of old uprobes kernel
module was removed from systemtap git repo by "PR30434 continuation:
Removed old uprobes, uprobes2 implementation, uprobes-inc.h & any
 mentions of CONFIG_UTRACE." it is good time for us to gid rid of it
too.

Signed-off-by: Victor Kamensky 
---
 meta/conf/distro/include/distro_alias.inc |  1 -
 meta/conf/distro/include/maintainers.inc  |  1 -
 .../systemtap/systemtap-uprobes_git.bb| 42 ---
 3 files changed, 44 deletions(-)
 delete mode 100644 meta/recipes-kernel/systemtap/systemtap-uprobes_git.bb

diff --git a/meta/conf/distro/include/distro_alias.inc 
b/meta/conf/distro/include/distro_alias.inc
index e43c0acd80..e6131c815a 100644
--- a/meta/conf/distro/include/distro_alias.inc
+++ b/meta/conf/distro/include/distro_alias.inc
@@ -317,7 +317,6 @@ DISTRO_PN_ALIAS:pn-sysprof = "Fedora=sysprof Debian=sysprof"
 DISTRO_PN_ALIAS:pn-systemd-compat-units = "Fedora=systemd Ubuntu=systemd"
 DISTRO_PN_ALIAS:pn-systemd-systemctl = "OE-Core"
 DISTRO_PN_ALIAS:pn-systemd-systemdctl = "Fedora=systemd Ubuntu=systemd"
-DISTRO_PN_ALIAS:pn-systemtap-uprobes = "Ubuntu=systemtap Debian=systemtap"
 DISTRO_PN_ALIAS:pn-sysvinit-inittab = "OE-Core"
 DISTRO_PN_ALIAS:pn-tar-replacement = "Fedora=tar Ubuntu=tar"
 DISTRO_PN_ALIAS:pn-tcf-agent = "Windriver 
upstream=http://www.eclipse.org/dsdp/tm/;
diff --git a/meta/conf/distro/include/maintainers.inc 
b/meta/conf/distro/include/maintainers.inc
index 2c59db56ac..445f797bcb 100644
--- a/meta/conf/distro/include/maintainers.inc
+++ b/meta/conf/distro/include/maintainers.inc
@@ -794,7 +794,6 @@ RECIPE_MAINTAINER:pn-systemd-serialgetty = "Chen Qi 
"
 RECIPE_MAINTAINER:pn-systemd-systemctl-native = "Chen Qi 
"
 RECIPE_MAINTAINER:pn-systemtap = "Victor Kamensky "
 RECIPE_MAINTAINER:pn-systemtap-native = "Victor Kamensky 
"
-RECIPE_MAINTAINER:pn-systemtap-uprobes = "Victor Kamensky 
"
 RECIPE_MAINTAINER:pn-sysvinit = "Ross Burton "
 RECIPE_MAINTAINER:pn-sysvinit-inittab = "Ross Burton "
 RECIPE_MAINTAINER:pn-taglib = "Anuj Mittal "
diff --git a/meta/recipes-kernel/systemtap/systemtap-uprobes_git.bb 
b/meta/recipes-kernel/systemtap/systemtap-uprobes_git.bb
deleted file mode 100644
index 3d35481bdc..00
--- a/meta/recipes-kernel/systemtap/systemtap-uprobes_git.bb
+++ /dev/null
@@ -1,42 +0,0 @@
-SUMMARY = "UProbes kernel module for SystemTap"
-HOMEPAGE = "https://sourceware.org/systemtap/;
-require systemtap_git.inc
-
-DEPENDS = "systemtap virtual/kernel"
-
-# On systems without CONFIG_UTRACE, this package is empty.
-ALLOW_EMPTY:${PN} = "1"
-
-inherit module-base gettext
-
-FILESEXTRAPATHS =. "${FILE_DIRNAME}/systemtap:"
-
-FILES:${PN} += "${datadir}/systemtap/runtime/uprobes"
-
-# Compile and install the uprobes kernel module on machines with utrace
-# support.  Note that staprun expects it in the systemtap/runtime directory,
-# not in /lib/modules.
-do_compile() {
-   if grep -q "CONFIG_UTRACE=y" ${STAGING_KERNEL_BUILDDIR}/.config
-   then
-   unset CFLAGS CPPFLAGS CXXFLAGS LDFLAGS CC LD CPP
-   oe_runmake CC="${KERNEL_CC}" LD="${KERNEL_LD}" \
-  AR="${KERNEL_AR}" OBJCOPY="${KERNEL_OBJCOPY}" \
-  STRIP="${KERNEL_STRIP}" \
-  -C ${STAGING_KERNEL_DIR} scripts
-   oe_runmake KDIR=${STAGING_KERNEL_DIR}   \
-  M="${S}/runtime/uprobes/" \
-  CC="${KERNEL_CC}" LD="${KERNEL_LD}" \
-  AR="${KERNEL_AR}" OBJCOPY="${KERNEL_OBJCOPY}" \
-  STRIP="${KERNEL_STRIP}" \
-  -C "${S}/runtime/uprobes/"
-   fi
-}
-
-do_install() {
-   if [ -e "${S}/runtime/uprobes/uprobes.ko" ]
-   then
-   install -d ${D}${datadir}/systemtap/runtime/uprobes/
-   install -m 0644 ${S}/runtime/uprobes/uprobes.ko 
${D}${datadir}/systemtap/runtime/uprobes/
-   fi
-}
-- 
2.31.1


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#191797): 
https://lists.openembedded.org/g/openembedded-core/message/191797
Mute This Topic: https://lists.openembedded.org/mt/102987516/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core] [PATCH 1/3] systemtap: upgrade 4.9 -> 5.0

[Victor Kamensky]

Similar to 4.9 release, in 5.0 release release-5.0 tag is off any branch,
release-5.0a tag e72dc118e563c645d93a1a2d771e8d90e8fec1ae is on the master
branch. Use this value for SRCREV and add catch up patch,
0001-prerelease-datestamp-fixes.patch, to bring source up to release-5.0 tag
level. Remove 0001-prerelease-datestamp-fixes.patch in next upgrade.

Signed-off-by: Victor Kamensky 
---
 .../0001-prerelease-datestamp-fixes.patch | 23 
 .../systemtap/0001-release-date-fix.patch | 37 ---
 .../systemtap/systemtap_git.inc   |  6 +--
 3 files changed, 26 insertions(+), 40 deletions(-)
 create mode 100644 
meta/recipes-kernel/systemtap/systemtap/0001-prerelease-datestamp-fixes.patch
 delete mode 100644 
meta/recipes-kernel/systemtap/systemtap/0001-release-date-fix.patch

diff --git 
a/meta/recipes-kernel/systemtap/systemtap/0001-prerelease-datestamp-fixes.patch 
b/meta/recipes-kernel/systemtap/systemtap/0001-prerelease-datestamp-fixes.patch
new file mode 100644
index 00..afdc10a3fe
--- /dev/null
+++ 
b/meta/recipes-kernel/systemtap/systemtap/0001-prerelease-datestamp-fixes.patch
@@ -0,0 +1,23 @@
+From cfc2c1d53924face11e3fab78ded61c359778eb9 Mon Sep 17 00:00:00 2001
+From: "Frank Ch. Eigler" 
+Date: Sat, 4 Nov 2023 12:19:59 -0400
+Subject: [PATCH] prerelease datestamp fixes
+
+Upstream-Status: Backport
+---
+ NEWS | 2 +-
+ 1 file changed, 1 insertion(+), 1 deletion(-)
+
+diff --git a/NEWS b/NEWS
+index c7f31e8b7..f0f9056c4 100644
+--- a/NEWS
 b/NEWS
+@@ -1,4 +1,4 @@
+-* What's new in version 5.0, 2023-11-03
++* What's new in version 5.0, 2023-11-04
+ 
+ - Performance improvements in uprobe registration and module startup.
+ 
+-- 
+2.31.1
+
diff --git 
a/meta/recipes-kernel/systemtap/systemtap/0001-release-date-fix.patch 
b/meta/recipes-kernel/systemtap/systemtap/0001-release-date-fix.patch
deleted file mode 100644
index 3dcb369ec3..00
--- a/meta/recipes-kernel/systemtap/systemtap/0001-release-date-fix.patch
+++ /dev/null
@@ -1,37 +0,0 @@
-From ebb424eee5599fcc131901c0d82d0bfc0d2f57ab Mon Sep 17 00:00:00 2001
-From: "Frank Ch. Eigler" 
-Date: Fri, 28 Apr 2023 13:51:27 -0400
-Subject: [PATCH] release date fix
-
-Upstream-Status: Backport

- NEWS   | 2 +-
- systemtap.spec | 2 +-
- 2 files changed, 2 insertions(+), 2 deletions(-)
-
-diff --git a/NEWS b/NEWS
-index 695bc990c..46859e304 100644
 a/NEWS
-+++ b/NEWS
-@@ -1,4 +1,4 @@
--* What's new in version 4.9, 2023-04-27
-+* What's new in version 4.9, 2023-04-28
- 
- - ISystemtap: the new interactive systemtap jupyter kernel. This provides
-   a simple GUI for writing/running Systemtap scripts in an easy, incremental
-diff --git a/systemtap.spec b/systemtap.spec
-index 59814f112..fd8023264 100644
 a/systemtap.spec
-+++ b/systemtap.spec
-@@ -1307,7 +1307,7 @@ exit 0
- 
- # PRERELEASE
- %changelog
--* Thu Apr 27 2023 Frank Ch. Eigler  - 4.9-1
-+* Fri Apr 28 2023 Frank Ch. Eigler  - 4.9-1
- - Upstream release, see wiki page below for detailed notes.
-   https://sourceware.org/systemtap/wiki/SystemTapReleases
- 
--- 
-2.40.1
-
diff --git a/meta/recipes-kernel/systemtap/systemtap_git.inc 
b/meta/recipes-kernel/systemtap/systemtap_git.inc
index de3750ff04..bc2a9ff434 100644
--- a/meta/recipes-kernel/systemtap/systemtap_git.inc
+++ b/meta/recipes-kernel/systemtap/systemtap_git.inc
@@ -1,13 +1,13 @@
 LICENSE = "GPL-2.0-only"
 LIC_FILES_CHKSUM = "file://COPYING;md5=b234ee4d69f5fce4486a80fdaf4a4263"
-SRCREV = "418f0a45ca4473491385b5c7eef777607bbdb3b7"
-PV = "4.9"
+SRCREV = "e72dc118e563c645d93a1a2d771e8d90e8fec1ae"
+PV = "5.0"
 
 SRC_URI = "git://sourceware.org/git/systemtap.git;branch=master;protocol=https 
\

file://0001-Do-not-let-configure-write-a-python-location-into-th.patch \
file://0001-Install-python-modules-to-correct-library-dir.patch \

file://0001-staprun-stapbpf-don-t-support-installing-a-non-root.patch \
-   file://0001-release-date-fix.patch \
+   file://0001-prerelease-datestamp-fixes.patch \
"
 
 COMPATIBLE_HOST = 
'(x86_64|i.86|powerpc|arm|aarch64|microblazeel|mips|riscv64).*-linux'
-- 
2.31.1


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#191795): 
https://lists.openembedded.org/g/openembedded-core/message/191795
Mute This Topic: https://lists.openembedded.org/mt/102987514/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core] [PATCH 2/3] systemtap: do not install uprobes and uprobes sources

[Victor Kamensky]

Addresses the following do_package_qa warning:

WARNING: systemtap-5.0-r0 do_package_qa: QA Issue: File 
/usr/share/systemtap/runtime/linux/uprobes/Makefile in package systemtap 
contains reference to TMPDIR
File /usr/share/systemtap/runtime/linux/uprobes/git_version.h in package 
systemtap contains reference to TMPDIR
File /usr/share/systemtap/runtime/linux/uprobes2/git_version.h in package 
systemtap contains reference to TMPDIR [buildpaths]

Signed-off-by: Victor Kamensky 
---
 ...ve-runtime-linux-uprobes-and-runtime.patch | 40 +++
 .../systemtap/systemtap_git.inc   |  1 +
 2 files changed, 41 insertions(+)
 create mode 100644 
meta/recipes-kernel/systemtap/systemtap/0001-Makefile.am-remove-runtime-linux-uprobes-and-runtime.patch

diff --git 
a/meta/recipes-kernel/systemtap/systemtap/0001-Makefile.am-remove-runtime-linux-uprobes-and-runtime.patch
 
b/meta/recipes-kernel/systemtap/systemtap/0001-Makefile.am-remove-runtime-linux-uprobes-and-runtime.patch
new file mode 100644
index 00..2d46a3962d
--- /dev/null
+++ 
b/meta/recipes-kernel/systemtap/systemtap/0001-Makefile.am-remove-runtime-linux-uprobes-and-runtime.patch
@@ -0,0 +1,40 @@
+From 5a01e28bd806326b2143e3e6bb28d4780c5d879d Mon Sep 17 00:00:00 2001
+From: Victor Kamensky 
+Date: Sun, 3 Dec 2023 18:40:05 -0800
+Subject: [PATCH] Makefile.am: remove runtime/linux/uprobes and
+ runtime/linux/uprobes2 install
+
+"PR30434 continuation:  Removed old uprobes, uprobes2 implementation,
+uprobes-inc.h & any mentions of CONFIG_UTRACE." commit removed uprobes,
+and uprobes2 sources and directories, but Makefile.am still tries to
+install them. In fact after failing to 'cd' into runtime/linux/uprobes
+directory it copies top level *.[ch] files into
+${prefix}/share/systemtap/runtime/linux/uprobes directory.
+
+The issue was caught by OpenEmbedded project do_package_qa checks.
+
+Signed-off-by: Victor Kamensky 
+
+Upstream-Status: Submitted 
[https://sourceware.org/pipermail/systemtap/2023q4/027880.html] 
+---
+ Makefile.am | 4 
+ 1 file changed, 4 deletions(-)
+
+diff --git a/Makefile.am b/Makefile.am
+index 5737c6b20..2ba896088 100644
+--- a/Makefile.am
 b/Makefile.am
+@@ -277,10 +277,6 @@ endif
+   do $(INSTALL_DATA) -D $$f 
$(DESTDIR)$(pkgdatadir)/runtime/softfloat/$$f; done)
+   (cd $(srcdir)/runtime/linux; for f in *.[ch]; \
+   do $(INSTALL_DATA) -D $$f 
$(DESTDIR)$(pkgdatadir)/runtime/linux/$$f; done)
+-  (cd $(srcdir)/runtime/linux/uprobes; for f in Makefile *.[ch]; \
+-  do $(INSTALL_DATA) -D $$f 
$(DESTDIR)$(pkgdatadir)/runtime/linux/uprobes/$$f; done)
+-  (cd $(srcdir)/runtime/linux/uprobes2; for f in *.[ch]; \
+-  do $(INSTALL_DATA) -D $$f 
$(DESTDIR)$(pkgdatadir)/runtime/linux/uprobes2/$$f; done)
+   (cd $(srcdir)/tapset; find . \( -name '*.stp' -o -name '*.stpm' -o 
-name README \) -print \
+   | while read f; do $(INSTALL_DATA) -D $$f 
$(DESTDIR)$(pkgdatadir)/tapset/$$f; done)
+   (cd $(srcdir)/testsuite/systemtap.examples; find . -type f -print \
+-- 
+2.31.1
+
diff --git a/meta/recipes-kernel/systemtap/systemtap_git.inc 
b/meta/recipes-kernel/systemtap/systemtap_git.inc
index bc2a9ff434..9fe85676fc 100644
--- a/meta/recipes-kernel/systemtap/systemtap_git.inc
+++ b/meta/recipes-kernel/systemtap/systemtap_git.inc
@@ -7,6 +7,7 @@ SRC_URI = 
"git://sourceware.org/git/systemtap.git;branch=master;protocol=https \

file://0001-Do-not-let-configure-write-a-python-location-into-th.patch \
file://0001-Install-python-modules-to-correct-library-dir.patch \

file://0001-staprun-stapbpf-don-t-support-installing-a-non-root.patch \
+   
file://0001-Makefile.am-remove-runtime-linux-uprobes-and-runtime.patch \
file://0001-prerelease-datestamp-fixes.patch \
"
 
-- 
2.31.1


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#191796): 
https://lists.openembedded.org/g/openembedded-core/message/191796
Mute This Topic: https://lists.openembedded.org/mt/102987515/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core] [PATCH] systemtap: upgrade 4.8 -> 4.9

[Victor Kamensky]

SystemTap 4.9 release went out with a little hick up: release date was not
set correctly and fix for it was added as 
ebb424eee5599fcc131901c0d82d0bfc0d2f57ab
commit and tagged as 'release-4.9'. Unfortunately by that time 'master'
branch already has moved on, and it turns out that now 'release-4.9' tag does
not belong to any branch. On other hand OE SRC_URI does complain about git
uri without branch. To deal with it we will use SRCREV set to
418f0a45ca4473491385b5c7eef777607bbdb3b7, commit one that precedes
ebb424eee5599fcc131901c0d82d0bfc0d2f57ab, and add separate patch for
ebb424eee5599fcc131901c0d82d0bfc0d2f57ab as 0001-release-date-fix.patch.

Note for any future version move: please remove 0001-release-date-fix.patch

Signed-off-by: Victor Kamensky 
---
 .../systemtap/0001-release-date-fix.patch | 37 +++
 .../systemtap/systemtap_git.inc   |  5 ++-
 2 files changed, 40 insertions(+), 2 deletions(-)
 create mode 100644 
meta/recipes-kernel/systemtap/systemtap/0001-release-date-fix.patch

diff --git 
a/meta/recipes-kernel/systemtap/systemtap/0001-release-date-fix.patch 
b/meta/recipes-kernel/systemtap/systemtap/0001-release-date-fix.patch
new file mode 100644
index 00..3dcb369ec3
--- /dev/null
+++ b/meta/recipes-kernel/systemtap/systemtap/0001-release-date-fix.patch
@@ -0,0 +1,37 @@
+From ebb424eee5599fcc131901c0d82d0bfc0d2f57ab Mon Sep 17 00:00:00 2001
+From: "Frank Ch. Eigler" 
+Date: Fri, 28 Apr 2023 13:51:27 -0400
+Subject: [PATCH] release date fix
+
+Upstream-Status: Backport
+---
+ NEWS   | 2 +-
+ systemtap.spec | 2 +-
+ 2 files changed, 2 insertions(+), 2 deletions(-)
+
+diff --git a/NEWS b/NEWS
+index 695bc990c..46859e304 100644
+--- a/NEWS
 b/NEWS
+@@ -1,4 +1,4 @@
+-* What's new in version 4.9, 2023-04-27
++* What's new in version 4.9, 2023-04-28
+ 
+ - ISystemtap: the new interactive systemtap jupyter kernel. This provides
+   a simple GUI for writing/running Systemtap scripts in an easy, incremental
+diff --git a/systemtap.spec b/systemtap.spec
+index 59814f112..fd8023264 100644
+--- a/systemtap.spec
 b/systemtap.spec
+@@ -1307,7 +1307,7 @@ exit 0
+ 
+ # PRERELEASE
+ %changelog
+-* Thu Apr 27 2023 Frank Ch. Eigler  - 4.9-1
++* Fri Apr 28 2023 Frank Ch. Eigler  - 4.9-1
+ - Upstream release, see wiki page below for detailed notes.
+   https://sourceware.org/systemtap/wiki/SystemTapReleases
+ 
+-- 
+2.40.1
+
diff --git a/meta/recipes-kernel/systemtap/systemtap_git.inc 
b/meta/recipes-kernel/systemtap/systemtap_git.inc
index afeb3ce2d2..de3750ff04 100644
--- a/meta/recipes-kernel/systemtap/systemtap_git.inc
+++ b/meta/recipes-kernel/systemtap/systemtap_git.inc
@@ -1,12 +1,13 @@
 LICENSE = "GPL-2.0-only"
 LIC_FILES_CHKSUM = "file://COPYING;md5=b234ee4d69f5fce4486a80fdaf4a4263"
-SRCREV = "b176afb2e49119ef844d193d27b0752a6d84fe8f"
-PV = "4.8"
+SRCREV = "418f0a45ca4473491385b5c7eef777607bbdb3b7"
+PV = "4.9"
 
 SRC_URI = "git://sourceware.org/git/systemtap.git;branch=master;protocol=https 
\

file://0001-Do-not-let-configure-write-a-python-location-into-th.patch \
file://0001-Install-python-modules-to-correct-library-dir.patch \

file://0001-staprun-stapbpf-don-t-support-installing-a-non-root.patch \
+   file://0001-release-date-fix.patch \
"
 
 COMPATIBLE_HOST = 
'(x86_64|i.86|powerpc|arm|aarch64|microblazeel|mips|riscv64).*-linux'
-- 
2.40.1


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#182406): 
https://lists.openembedded.org/g/openembedded-core/message/182406
Mute This Topic: https://lists.openembedded.org/mt/99357361/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [OE-core] [PATCH v2 0/5] systemtap improvements

[Victor Kamensky]

Hi Yann and Richard,

On Fri, Nov 13, 2020 at 5:59 AM Yann Dirson 
wrote:
>
> Oh I did not notice, shouldn't we be discussing this on-list ?

Very sorry about this. I did not notice that I've unintentionally removed
the mailing list.
Added it back.

I see that Richard added your patches to master-next. But I don't think
they should be
merged "as is". IMO we need to work more on them.

Richard, could you please hold off merging them?

Other thing that AUT done automatic upgrade to systemtap-4.4 and Zang have
sent patch
for it. IMO let's first merge 4.4 while we sort out what part of this
series should go in.

Let's go about slowly one issue at a time.

More responses inline.

> Another question: I can't really see what it brings, to be able to
disable the "monitor" feature.
> What should we write about this feature ?
>
> Le ven. 13 nov. 2020 à 11:57, Yann Dirson  a
écrit :
>>
>>
>>
>> Le jeu. 12 nov. 2020 à 21:11, Victor Kamensky 
a écrit :
>>>
>>> Hi Yann,
>>>
>>> I am not sure that I understand purpose of the series.
>>
>>
>> The goal was to be able to include a precompiled probe in a readonly
image, and to get
>> it loaded during execution of a boot script (I was hunting a but that
only shows during
>> target startup).

It is an interesting feature, do you plan to merge precompiled probes in
production image, or
you want to add one for some debug case. If it is latter one probably there
are ways to do
it just by adding file built crosstap just as additional payload to the
image.

In developper case I personally deal with it by stopping system boot very
early i.e using
systemd "emergency" parameter passed as kernel boot option, dropping into
the shell
mounting some storage with precompiled probe, copy and activate probe and
exiting
emergency shell.

Also if we do with supporting such a feature one should be able to build
probe not only for
kernel but for the whole user-land. Does your proposed patch series support
probes for user-land
too? If so, how do you do that: chicken and egg problem - to build
user-land probe one need
complete sysroot of target, but since probe should be include in the image,
how do one
build sysroot first.

>>
>> It happens that I had not discovered the crosstap tool when I started
this, but it does not seem
>> to help much in my use-case.
>>
>> My approach was naturally integrated into the build, packaging the
cross-build of the .stp module,
>> finally giving birth to the systemtap.bbclass you can find in v1 of this
series (which I'm here postponing
>> to a further series).
>>
>> I also wanted to avoid installing unused stuff on target, and starting
disabling some PACKAGECONFIG
>> items, finding disabling some caused problems.
>>
>>>
>>> The current way to include SystemTap minimal runtime only into target
image
>>> that allow to run compiled on host scripts is to use
>>>
>>> PACKAGECONFIG_pn-systemtap =  openembedded-core@lists.openembedded.org
"monitor"

>>
>>
>> OK, for some reason I had missed that it does indeed achieve that.
Maybe we should add more doc about that.
>>
>> The split of a -runtime package, however, allows to still build the
"translator" part, not have it installed in the image
>> shipped on product, but still have the ability to install it from the
feed while developing, should someone prefer
>> this to cross-building the probes.  Not a big deal if you prefer not to
do that however, my own use-case would be
>> covered by the PACKAGECONFIG="monitor".

Yes, I was thinking about this use case as well. But it is quite esoteric
for OE. Furthermore,
it kind of contradicts current PACKAGECONFIG structure. If we want to go
this route we need
to reshuffle the whole thing in a different way. I.e either use
PACKAGECONFIG to
control the content systemtap package or use subpackages. Using both looks
a bit convoluted.

>>
>> But as state above it does not build on master/gatesgarth/dunfell
because of the problem fixed by the "fix install when python3-probes is
disabled in PACKAGECONFIG"
>> patch in this series.
>>
>> Furthermore, this minimal package with just runtime support still pulls
python3-core without a need for it.
>> This is coming from "inherit distutils3-base", and we should probably
not pull this one unless "python3-probes"
>> is enabled.  I had missed this one, will add another patch for this.
>>
>> BTW, does it make any sense to select "python3-probes" but not
"translator" ?

I don't quite yet follow this issue and need to play more before answering
it. Could you
please post separate patch only for this?

Thanks,
Victor


On Fri, N

Re: [OE-core] [PATCH v3] make: 4.2.1 -> 4.3

[Victor Kamensky (kamensky) via Openembedded-core]

s not load of -1 into a0 before jumping to setresuid system call.

In disassembly at certain point one can see:

   0x00fff7edde1c <+860>:   li  a0,127

It seems either internal_syscall3 is wrong or compiler does not
process it correctly ... quite bad ...

Looking at make make-4.3/src/job.c
--

It seems that posix_spawn would be used only if USE_POSIX_SPAWN defined.

Test for USE_POSIX_SPAWN is in configure.ac
---

>From cat -n make-4.3/configure.ac

   382  AS_IF([test "$make_cv_posix_spawn" = yes],
   383AC_CACHE_CHECK([for posix_spawn that fails synchronously],
   384  [make_cv_synchronous_posix_spawn],
   385  [make_cv_synchronous_posix_spawn=no
   386   AC_RUN_IFELSE([AC_LANG_SOURCE([[
   387 #include 
   388 #include 
   389  
   390 extern char **environ;
   391  
   392 int main() {
   393   char* path = strdup("./non-existent");
   394   char *argv[[2]];
   395   argv[[0]] = path;
   396   argv[[1]] =  0;
   397   return posix_spawn(0, path, 0, 0, argv, environ);
   398 }]])],
   399 [make_cv_synchronous_posix_spawn=no],
   400 [make_cv_synchronous_posix_spawn=yes],
   401 [make_cv_synchronous_posix_spawn="no (cross-compiling)"])]))
   402  
   403  
AS_CASE([/$user_posix_spawn/$make_cv_posix_spawn/$make_cv_synchronous_posix_spawn/],
   404[*/no/*], [make_cv_posix_spawn=no],
   405[AC_DEFINE(USE_POSIX_SPAWN, 1, [Define to 1 to use posix_spawn().])
   406])

Note USE_POSIX_SPAWN code was not present in make-4.2.1
---

Workaround
---

In case of mips disable use of posix_spawn through ac_cv_func_posix_spawn=no
configure option.

Testing
---

Tested 'bitbake core-image-sato-sdk -c testimage' all tested are passed.

Thanks,
Victor


From: Richard Purdie 
Sent: Thursday, February 27, 2020 11:45 PM
To: Jens Rehsack; openembedded-core@lists.openembedded.org
Cc: Bruce Ashfield; Victor Kamensky (kamensky)
Subject: Re: [OE-core] [PATCH v3] make: 4.2.1 -> 4.3

On Mon, 2020-02-24 at 15:39 +0100, Jens Rehsack wrote:
> Announcement: 
> https://lists.gnu.org/archive/html/bug-make/2020-01/msg00057.html
>
> 1) Remove upstream provided patches 
> 0001-glob-Do-not-assume-glibc-glob-internals.patch
>and 0002-glob-Do-not-assume-glibc-glob-internals.patch.
>
> 2) License has been changed to GPLv3 only
>
> 3) Important bug-fix is
>* https://lists.gnu.org/archive/html/bug-make/2018-09/msg6.html
>
> 4) Backward-incompatibilities:
>* Number signs (#) appearing inside a macro reference or function 
> invocation
>  no longer introduce comments and should not be escaped with backslashes
>* Previously appending using '+=' to an empty variable would result in a 
> value
>  starting with a space.  Now the initial space is only added if the 
> variable
>  already contains some value.  Similarly, appending an empty string does 
> not
>  add a trailing space.
>
> Fix incompatibility issues between gnulib bundled with updated make fix issues
> in w32 compat sources.
>
> Signed-off-by: Jens Rehsack https://autobuilder.yoctoproject.org/typhoon/#/builders/60/builds/1612
https://autobuilder.yoctoproject.org/typhoon/#/builders/102/builds/315
https://autobuilder.yoctoproject.org/typhoon/#/builders/74/builds/1616

Cheers,

Richard

From 088ebd7b1c50babcf131cb0364cf8841b697863b Mon Sep 17 00:00:00 2001
From: Victor Kamensky 
Date: Fri, 28 Feb 2020 22:26:11 -0800
Subject: [PATCH] make: disable use of posix_spawn on mips

After make-4.3 migration child_execute_job function started
using posix_spawn function, which happens to be broken on mips.

It manifests itself as when make executed by root, it switches
real user id to wrong value because of some issues with direct
setresuid system call done in glibc __spawni_child function
through inline assemble and/or gcc compiling it produces wrong
code. I.e instead of passing -1 posix_spawn function incorrectly
passes 127 as ruid. Subsequently job started by make can fail
with permission issue because they run under wrong user.

For now workaround is used by explicitly disabling posix_spawn
call use by make on mips through configure variable.

Signed-off-by: Victor Kamensky 
---
 meta/recipes-devtools/make/make_4.3.bb | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/meta/recipes-devtools/make/make_4.3.bb b/meta/recipes-devtools/make/make_4.3.bb
index 70caf0ae16..5fa7059169 100644
--- a/meta/recipes-devtools/make/make_4.3.bb
+++ b/meta/recipes-devtools/make/make_4.3.bb
@@ -12,6 +12,9 @@ SRC_URI += "\
 
 EXTRA_OECONF += "--without-guile"
 
+EXTRA_OECONF_append_mips=" ac_cv_func_posix_spawn=no"
+EXTRA_OECONF_append_mips64=" ac_cv_func_posix_spawn=no"
+
 SRC_URI[md5sum] = "d5c40e7bd1e97a7404f5d3be982f479a"
 SRC_URI[sha256sum] = "de1a441c4edf952521db30bfca80baae86a0ff1acd0a00402999344f04c45e82"
 
-- 
2.21.1

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH] make: disable use of posix_spawn on mips

[Victor Kamensky via Openembedded-core]

After make-4.3 migration child_execute_job function started
using posix_spawn function, which happens to be broken on mips.

It manifests itself as when make executed by root, it switches
real user id to wrong value because of some issues with direct
setresuid system call done in glibc __spawni_child function
through inline assemble and/or gcc compiling it produces wrong
code. I.e instead of passing -1 posix_spawn function incorrectly
passes 127 as ruid. Subsequently job started by make can fail
with permission issue because they run under wrong user.

For now workaround is used by explicitly disabling posix_spawn
call use by make on mips through configure variable.

Signed-off-by: Victor Kamensky 
---
 meta/recipes-devtools/make/make_4.3.bb | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/meta/recipes-devtools/make/make_4.3.bb 
b/meta/recipes-devtools/make/make_4.3.bb
index 70caf0ae16..5fa7059169 100644
--- a/meta/recipes-devtools/make/make_4.3.bb
+++ b/meta/recipes-devtools/make/make_4.3.bb
@@ -12,6 +12,9 @@ SRC_URI += "\
 
 EXTRA_OECONF += "--without-guile"
 
+EXTRA_OECONF_append_mips=" ac_cv_func_posix_spawn=no"
+EXTRA_OECONF_append_mips64=" ac_cv_func_posix_spawn=no"
+
 SRC_URI[md5sum] = "d5c40e7bd1e97a7404f5d3be982f479a"
 SRC_URI[sha256sum] = 
"de1a441c4edf952521db30bfca80baae86a0ff1acd0a00402999344f04c45e82"
 
-- 
2.21.1

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 13/42] systemtap: upgrade 4.0 -> 4.1

[Victor Kamensky (kamensky) via Openembedded-core]

Hi Alexander,

I am curios how much did you test it? I had the same patch for
a while, waited to post it till proper regressions are done.

Thanks,
Victor


From: openembedded-core-boun...@lists.openembedded.org 
 on behalf of Alexander 
Kanavin 
Sent: Monday, May 20, 2019 7:10 AM
To: openembedded-core@lists.openembedded.org
Subject: [OE-core] [PATCH 13/42] systemtap: upgrade 4.0 -> 4.1

Signed-off-by: Alexander Kanavin 
---
 meta/recipes-kernel/systemtap/systemtap_git.inc | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/meta/recipes-kernel/systemtap/systemtap_git.inc 
b/meta/recipes-kernel/systemtap/systemtap_git.inc
index 54de7226608..c5348b3204b 100644
--- a/meta/recipes-kernel/systemtap/systemtap_git.inc
+++ b/meta/recipes-kernel/systemtap/systemtap_git.inc
@@ -1,7 +1,7 @@
 LICENSE = "GPLv2"
 LIC_FILES_CHKSUM = "file://COPYING;md5=b234ee4d69f5fce4486a80fdaf4a4263"
-SRCREV = "90ff34ac0506f0d5440393a4c78565f6aaf9b726"
-PV = "4.0"
+SRCREV = "984d6d1696ed06626b07cb65ab55d6ae0ece1131"
+PV = "4.1"

 SRC_URI = "git://sourceware.org/git/systemtap.git \
file://configure-allow-to-disable-libvirt.patch \
--
2.17.1

--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 4/5] linux-yocto: Fix systemtap issue on armv7

[Victor Kamensky via Openembedded-core]

On Sun, 10 Mar 2019, Bruce Ashfield wrote:


On Sun, Mar 10, 2019 at 11:12 PM Richard Purdie
 wrote:


Testing stap on armv7 machines was failing due to intermixing of thumb/arm
instructions. Patch the kernel to always use the v7 march options since
we know our gcc versions support it to avoid the failure and allow
systemtap to work.


I'd rather just merge this into the kernel directly. I can do a quick
merge and bump
of the SRCREV. If you send the patch against linux-yocto, that makes it easier
for me to grab and integrate it, versus having to dig it out of here
(since my access
is pretty limited at the moment).


BTW please note I posted on ARM linux kernel mailing list
IMO more appropriate and possibly upstreamable fix for the same issue

http://lists.infradead.org/pipermail/linux-arm-kernel/2019-March/637809.html

It seems quite urgent so Richard's fix might work as well as short term.

More inline.


Bruce



[YOCTO #13153]

Signed-off-by: Richard Purdie 
---
 ...0001-arm-Makefile-Fix-systemtap-4.19.patch | 60 +++
 .../0001-arm-Makefile-Fix-systemtap.patch | 60 +++
 meta/recipes-kernel/linux/linux-yocto_4.19.bb |  3 +-
 meta/recipes-kernel/linux/linux-yocto_5.0.bb  |  3 +-
 4 files changed, 124 insertions(+), 2 deletions(-)
 create mode 100644 
meta/recipes-kernel/linux/linux-yocto/0001-arm-Makefile-Fix-systemtap-4.19.patch
 create mode 100644 
meta/recipes-kernel/linux/linux-yocto/0001-arm-Makefile-Fix-systemtap.patch

diff --git 
a/meta/recipes-kernel/linux/linux-yocto/0001-arm-Makefile-Fix-systemtap-4.19.patch
 
b/meta/recipes-kernel/linux/linux-yocto/0001-arm-Makefile-Fix-systemtap-4.19.patch
new file mode 100644
index 000..53bd53b276c
--- /dev/null
+++ 
b/meta/recipes-kernel/linux/linux-yocto/0001-arm-Makefile-Fix-systemtap-4.19.patch
@@ -0,0 +1,60 @@
+From c2995494e311c113177db50ff140cebd94fd4011 Mon Sep 17 00:00:00 2001
+From: Richard Purdie 
+Date: Sun, 10 Mar 2019 06:43:15 +
+Subject: [PATCH] arm/Makefile: Fix systemtap
+
+Currently systemtap fails to operate correctly on armv7 systems such as 
beaglebone and
+soon, qemuarm.
+
+
+root@qemuarm:/usr/src/kernel# env -uARCH -uKBUILD_EXTMOD -uCROSS_COMPILE 
-uKBUILD_IMAGE -uKCONFIG_CONFIG -uINSTALL_PATH -uLD_LIBRARY_PATH 
PATH=/usr/bin:/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin 
make -C /lib/modules/4.19.19-yocto-standard/build M=/tmp/staptcNU6M modules 
CONFIG_DEBUG_INFO= CONFIG_STACK_VALIDATION= ARCH=arm stap_4321_src.i 
--no-print-directory -j2 V=1
+test -e include/generated/autoconf.h -a -e include/config/auto.conf || (   
\
+echo >&2;  \
+echo >&2 "  ERROR: Kernel configuration is invalid.";  \
+echo >&2 " include/generated/autoconf.h or include/config/auto.conf are 
missing.";\
+echo >&2 " Run 'make oldconfig && make prepare' on kernel src to fix 
it."; \
+echo >&2 ; \
+/bin/false)
+mkdir -p /tmp/staptcNU6M/.tmp_versions ; rm -f /tmp/staptcNU6M/.tmp_versions/*
+make -f ./scripts/Makefile.build obj=/tmp/staptcNU6M
+(cat /dev/null;   echo kernel//tmp/staptcNU6M/stap_4321.ko;) > 
/tmp/staptcNU6M/modules.order
+  gcc -Wp,-MD,/tmp/staptcNU6M/.stap_4321_src.o.d  -nostdinc -isystem 
/usr/lib/gcc/arm-poky-linux-gnueabi/8.3.0/include -I./arch/arm/include 
-I./arch/arm/include/generated  -I./include -I./arch/arm/include/uapi 
-I./arch/arm/include/generated/uapi -I./include/uapi -I./include/generated/uapi 
-include ./include/linux/kconfig.h -include ./include/linux/compiler_types.h 
-D__KERNEL__ -mlittle-endian -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs 
-fno-strict-aliasing -fno-common -fshort-wchar 
-Werror-implicit-function-declaration -Wno-format-security -std=gnu89 -fno-PIE 
-DCC_HAVE_ASM_GOTO -fno-dwarf2-cfi-asm -fno-omit-frame-pointer -mapcs 
-mno-sched-prolog -fno-ipa-sra -mabi=aapcs-linux -mfpu=vfp -funwind-tables 
-marm -Wa,-mno-warn-deprecated -D__LINUX_ARM_ARCH__=7 -march=armv5t 
-Wa,-march=armv7-a -msoft-float -Uarm -fno-delete-null-pointer-checks 
-Wno-frame-address -Wno-format-truncation -Wno-format-overflow 
-Wno-int-in-bool-context -Os -Wno-maybe-uninitialized --param=allow-s

t

ore-data-races=0 -Wframe-larger-than=1024 -fstack-protector-strong -Wno-unused-but-set-variable 
-Wno-unused-const-variable -fno-omit-frame-pointer -fno-optimize-sibling-calls -fno-var-tracking-assignments 
-pg -Wdeclaration-after-statement -Wno-pointer-sign -Wno-stringop-truncation -fno-strict-overflow 
-fno-merge-all-constants -fmerge-constants -fno-stack-check -fconserve-stack -Werror=implicit-int 
-Werror=strict-prototypes -Werror=date-time -Werror=incompatible-pointer-types -Werror=designated-init 
-fmacro-prefix-map=./= -Wno-packed-not-aligned -Iinclude2/asm/mach-default 
-I/lib/modules/4.19.19-yocto-standard/build -include /tmp/staptcNU6M/stapconf_4321.h -D 
"STP_NO_VELREL_CHECK" -freorder-blocks -fasynchronous-unwind-tables 

Re: [OE-core] [PATCH] sysklogd: add alternatives for klogd and syslogd

[Victor Kamensky via Openembedded-core]

Hi Mark,

On Mon, 29 Oct 2018, Mark Hatle wrote:


On 10/26/18 2:55 AM, Victor Kamensky wrote:



On Fri, 26 Oct 2018, Markus Lehtonen wrote:


'why would you want to have multiple, alternative
syslog daemons on the system? Wouldn't a better
fix be to move the syslogd and klogd symlinks to the
busybox-syslog package? It doesn't seem to make much
sense to have a syslogd binary on the system without
an init script(?)

The original problem really was related to managing
init scripts with alternatives (and I think that
problem still persists, and, it shouldn't be done).
So, using alternatives for the binaries wouldn't
cause any problems. The question is just why to
do this, instead of "fixing" the (binary) package
content so that existing rconflicts would handle it.


Thank you, Markus. Please see ChenQi reply on the
thread. I has already been fixed by Richard in the
way you suggested.

My problem was because similar fix was not present
in meta-selinux busybox.bbappend ... meta-selinux practically
has a copy of alternative handling for busybox
because of SELinux labeling requirements and it was
negating Richard's change.


As a maintainer for meta-selinux, if meta-selinux is broken.. it's
meta-selinux's fault.. NOT oe-core.
oe-core must work for itself first.

Fix OE-core properly and the other layers will follow.


Please see

http://lists.openembedded.org/pipermail/openembedded-core/2018-October/157078.html

oe-core is fine: ef11c54ba9. In order to follow
meta-selinux needs something like this to avoid
'update-alternatives: Error:' mentioned on the thead:


From dd75b2ca77b3bfcb14030c353e6b2bfc7df8dfec Mon Sep 17 00:00:00 2001

From: Victor Kamensky 
Date: Mon, 29 Oct 2018 11:16:09 -0700
Subject: [PATCH] busybox: Put klogd/syslogd alternative links in syslog
 package

Port "ef11c54ba9 busybox: Put klogd/syslogd alternative links in
syslog package" from oe-core to meta-selinux copy of
update-alternative handling in busybox.bbappend

Signed-off-by: Victor Kamensky 
---
 recipes-core/busybox/busybox_selinux.inc | 6 +-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/recipes-core/busybox/busybox_selinux.inc 
b/recipes-core/busybox/busybox_selinux.inc
index df7c117..b8cfcd7 100644
--- a/recipes-core/busybox/busybox_selinux.inc
+++ b/recipes-core/busybox/busybox_selinux.inc
@@ -37,7 +37,11 @@ python create_sh_wrapper_reset_alternative_vars () {
 # Match coreutils
 if alt_name == '[':
 alt_name = 'lbracket'
-d.appendVar('ALTERNATIVE_%s' % (pn), ' ' + alt_name)
+if alt_name == 'klogd' or alt_name == 'syslogd':
+d.appendVar('ALTERNATIVE_%s-syslog' % (pn), ' ' + alt_name)
+else:
+d.appendVar('ALTERNATIVE_%s' % (pn), ' ' + alt_name)
+
 d.setVarFlag('ALTERNATIVE_LINK_NAME', alt_name, alt_link_name)
 if os.path.exists(alt_wppath_dest):
 d.setVarFlag('ALTERNATIVE_TARGET', alt_name, alt_wppath)
--
2.14.4

I don't know yet what mailing list I should use for meta-selinux. As
time permits I'll try to figure out and post above patch there, if
it is still needed.

Thanks,
Victor


--Mark


So it is sorted out now.

Thanks,
Victor


Cheers,
 Markus

On 26/10/2018, 7.55, "Victor Kamensky"  wrote:

   Otherwise when used in presense of busybox that provides
   its own version of klogd and syslogd, image packaging
   complains that klogd exists and it is not syymbolic link.
   Failure happens only if image packaging script install
   sysklogd package first followed by installtion of busybox
   package. If during packaging reverse installtion order
   happens, busybox first followed by sysklogd, packaging
   succeed.

   Note this fix along with recently committed
   55ba9dc1f8 sysklogd: Re-enable alternatives for syslogd.8 man page
   effectively reverts this commit
   988aad01b2 sysklogd: don't use update-alternatives

   Signed-off-by: Victor Kamensky 
   ---
   Hi Guys,

   Here is more details. Example of failure that I observe:

   update-alternatives: Error: not linking 
/home/wd8/oe/20181021/build/tmp-glibc/work/intel_corei7_64-oe-linux/kdevel-console-devel-image/1.0-r0/rootfs/sbin/klogd
 to /usr/lib/busybox/sbin/klogd since 
/home/wd8/oe/20181021/build/tmp-glibc/work/intel_corei7_64-oe-linux/kdevel-console-devel-image/1.0-r0/rootfs/sbin/klogd
 exists and is not a link

   Also 988aad01b2 says:

  > Using update-alternatives for managing init scripts has proved to be
  > problematic. And, sysklogd rconflicts with other syslog daemons so there
  > is no point in using update-alternatives from this perspective, either.

   I am not sure why "managing init scripts has proved to be problematic" and
   syslogd and klogd are not really init script per se, aren't they? Also
   klogd and syslogd actually come from busybox, not busybox-syslog as listed
   in sysklogd RCONFLICTS. Maybe it what has changed since 988aad01b2

[OE-core] [PATCH] meson: map powerpc64 TARGET_ARCH to ppc64 for the cross file

[Victor Kamensky via Openembedded-core]

Meson uses 'ppc64' for 64 bit powerpc. Issue came up while
building systemd for MACHINE that uses ppc64e5500 tune.

Signed-off-by: Victor Kamensky 
---
BTW very nice diagnostic message through
https://wiki.yoctoproject.org/wiki/Meson/UnknownCPU, that led directly 
to the fix

 meta/classes/meson.bbclass | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/meta/classes/meson.bbclass b/meta/classes/meson.bbclass
index 7e63e12588..3cbdcf18c2 100644
--- a/meta/classes/meson.bbclass
+++ b/meta/classes/meson.bbclass
@@ -52,6 +52,8 @@ def meson_cpu_family(var, d):
 arch = d.getVar(var)
 if arch == 'powerpc':
 return 'ppc'
+elif arch == 'powerpc64':
+return 'ppc64'
 elif arch == 'mipsel':
 return 'mips'
 elif re.match(r"i[3-6]86", arch):
-- 
2.14.4

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH] sysklogd: add alternatives for klogd and syslogd

[Victor Kamensky via Openembedded-core]

On Fri, 26 Oct 2018, Markus Lehtonen wrote:


'why would you want to have multiple, alternative
syslog daemons on the system? Wouldn't a better
fix be to move the syslogd and klogd symlinks to the
busybox-syslog package? It doesn't seem to make much
sense to have a syslogd binary on the system without
an init script(?)

The original problem really was related to managing
init scripts with alternatives (and I think that
problem still persists, and, it shouldn't be done).
So, using alternatives for the binaries wouldn't
cause any problems. The question is just why to
do this, instead of "fixing" the (binary) package
content so that existing rconflicts would handle it.


Thank you, Markus. Please see ChenQi reply on the
thread. I has already been fixed by Richard in the
way you suggested.

My problem was because similar fix was not present
in meta-selinux busybox.bbappend ... meta-selinux practically
has a copy of alternative handling for busybox
because of SELinux labeling requirements and it was
negating Richard's change.

So it is sorted out now.

Thanks,
Victor


Cheers,
 Markus

On 26/10/2018, 7.55, "Victor Kamensky"  wrote:

   Otherwise when used in presense of busybox that provides
   its own version of klogd and syslogd, image packaging
   complains that klogd exists and it is not syymbolic link.
   Failure happens only if image packaging script install
   sysklogd package first followed by installtion of busybox
   package. If during packaging reverse installtion order
   happens, busybox first followed by sysklogd, packaging
   succeed.

   Note this fix along with recently committed
   55ba9dc1f8 sysklogd: Re-enable alternatives for syslogd.8 man page
   effectively reverts this commit
   988aad01b2 sysklogd: don't use update-alternatives

   Signed-off-by: Victor Kamensky 
   ---
   Hi Guys,

   Here is more details. Example of failure that I observe:

   update-alternatives: Error: not linking 
/home/wd8/oe/20181021/build/tmp-glibc/work/intel_corei7_64-oe-linux/kdevel-console-devel-image/1.0-r0/rootfs/sbin/klogd
 to /usr/lib/busybox/sbin/klogd since 
/home/wd8/oe/20181021/build/tmp-glibc/work/intel_corei7_64-oe-linux/kdevel-console-devel-image/1.0-r0/rootfs/sbin/klogd
 exists and is not a link

   Also 988aad01b2 says:

   > Using update-alternatives for managing init scripts has proved to be
   > problematic. And, sysklogd rconflicts with other syslog daemons so there
   > is no point in using update-alternatives from this perspective, either.

   I am not sure why "managing init scripts has proved to be problematic" and
   syslogd and klogd are not really init script per se, aren't they? Also
   klogd and syslogd actually come from busybox, not busybox-syslog as listed
   in sysklogd RCONFLICTS. Maybe it what has changed since 988aad01b2.
   busybox-syslog now contains only init script for syslog and its
   configuration.

   Adding Markus for further comments.

meta/recipes-extended/sysklogd/sysklogd.inc | 5 +
1 file changed, 5 insertions(+)

   diff --git a/meta/recipes-extended/sysklogd/sysklogd.inc 
b/meta/recipes-extended/sysklogd/sysklogd.inc
   index f151dd87f7..4393a39180 100644
   --- a/meta/recipes-extended/sysklogd/sysklogd.inc
   +++ b/meta/recipes-extended/sysklogd/sysklogd.inc
   @@ -60,9 +60,14 @@ FILES_${PN} += 
"${@bb.utils.contains('DISTRO_FEATURES','systemd','${exec_prefix}

ALTERNATIVE_PRIORITY = "100"

   +ALTERNATIVE_${PN} = "syslogd klogd"
   +
ALTERNATIVE_${PN}-doc = "syslogd.8"
ALTERNATIVE_LINK_NAME[syslogd.8] = "${mandir}/man8/syslogd.8"

   +ALTERNATIVE_LINK_NAME[syslogd] = "${base_sbindir}/syslogd"
   +ALTERNATIVE_LINK_NAME[klogd] = "${base_sbindir}/klogd"
   +
pkg_prerm_${PN} () {
if test "x$D" = "x"; then
if test "$1" = "upgrade" -o "$1" = "remove"; then
   --
   2.17.2




-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH] sysklogd: add alternatives for klogd and syslogd

[Victor Kamensky via Openembedded-core]

Hi Chen,

On Fri, 26 Oct 2018, ChenQi wrote:


On 10/26/2018 12:53 PM, Victor Kamensky via Openembedded-core wrote:

Otherwise when used in presense of busybox that provides
its own version of klogd and syslogd, image packaging
complains that klogd exists and it is not syymbolic link.
Failure happens only if image packaging script install
sysklogd package first followed by installtion of busybox
package.


Hi Victor,

I think this problem has been fixed by the following commit.

"""
commit ef11c54ba99af261a70ec31091216cdd1556da24
Author: Richard Purdie 
Date:   Wed Sep 5 17:39:31 2018 +0100

   busybox: Put klogd/syslogd alternative links in syslog package

   Currently these are in ${PN} and ${PN}-syslog may get replaced by
   other packages but update-alternatives would error in the postinst
   if other files were installed first. Avoid the problems by putting
   the links in the correct package.

   Signed-off-by: Richard Purdie 
"""

For the current status, the update-alternatives operations for syslogd and 
klogd should be in the busybox-syslog package, which conflicts with sysklogd 
package.


Could you please help check this?


Thank you! That helps. I see it now. Apprently busybox.bbappend
from meta-selinux layer:

http://git.yoctoproject.org/cgit/cgit.cgi/meta-selinux/tree/recipes-core/busybox/busybox_selinux.inc

is effectively undoing effect of above commit in my workspace.

meta-selinux is doing it because of

http://git.yoctoproject.org/cgit/cgit.cgi/meta-selinux/commit/?id=521ca9c9cf370840e9f8c808a7955aa5da7c356e

It seems changes similar to above Richard's commit needed to
be applied there as well. After applying that to busybox_selinux.inc
in meta-selinux my problem goes away.

No need for proposed below patch.

Thanks,
Victor


Best Regards,
Chen Qi


  If during packaging reverse installtion order
happens, busybox first followed by sysklogd, packaging
succeed.

Note this fix along with recently committed
55ba9dc1f8 sysklogd: Re-enable alternatives for syslogd.8 man page
effectively reverts this commit
988aad01b2 sysklogd: don't use update-alternatives

Signed-off-by: Victor Kamensky 
---
Hi Guys,

Here is more details. Example of failure that I observe:

update-alternatives: Error: not linking 
/home/wd8/oe/20181021/build/tmp-glibc/work/intel_corei7_64-oe-linux/kdevel-console-devel-image/1.0-r0/rootfs/sbin/klogd 
to /usr/lib/busybox/sbin/klogd since 
/home/wd8/oe/20181021/build/tmp-glibc/work/intel_corei7_64-oe-linux/kdevel-console-devel-image/1.0-r0/rootfs/sbin/klogd 
exists and is not a link


Also 988aad01b2 says:


Using update-alternatives for managing init scripts has proved to be
problematic. And, sysklogd rconflicts with other syslog daemons so there
is no point in using update-alternatives from this perspective, either.

I am not sure why "managing init scripts has proved to be problematic" and
syslogd and klogd are not really init script per se, aren't they? Also
klogd and syslogd actually come from busybox, not busybox-syslog as listed
in sysklogd RCONFLICTS. Maybe it what has changed since 988aad01b2.
busybox-syslog now contains only init script for syslog and its
configuration.

Adding Markus for further comments.

  meta/recipes-extended/sysklogd/sysklogd.inc | 5 +
  1 file changed, 5 insertions(+)

diff --git a/meta/recipes-extended/sysklogd/sysklogd.inc 
b/meta/recipes-extended/sysklogd/sysklogd.inc

index f151dd87f7..4393a39180 100644
--- a/meta/recipes-extended/sysklogd/sysklogd.inc
+++ b/meta/recipes-extended/sysklogd/sysklogd.inc
@@ -60,9 +60,14 @@ FILES_${PN} += 
"${@bb.utils.contains('DISTRO_FEATURES','systemd','${exec_prefix}

ALTERNATIVE_PRIORITY = "100"
  +ALTERNATIVE_${PN} = "syslogd klogd"
+
  ALTERNATIVE_${PN}-doc = "syslogd.8"
  ALTERNATIVE_LINK_NAME[syslogd.8] = "${mandir}/man8/syslogd.8"
  +ALTERNATIVE_LINK_NAME[syslogd] = "${base_sbindir}/syslogd"
+ALTERNATIVE_LINK_NAME[klogd] = "${base_sbindir}/klogd"
+
  pkg_prerm_${PN} () {
if test "x$D" = "x"; then
if test "$1" = "upgrade" -o "$1" = "remove"; then





--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH] sysklogd: add alternatives for klogd and syslogd

[Victor Kamensky via Openembedded-core]

Otherwise when used in presense of busybox that provides
its own version of klogd and syslogd, image packaging
complains that klogd exists and it is not syymbolic link.
Failure happens only if image packaging script install
sysklogd package first followed by installtion of busybox
package. If during packaging reverse installtion order
happens, busybox first followed by sysklogd, packaging
succeed.

Note this fix along with recently committed
55ba9dc1f8 sysklogd: Re-enable alternatives for syslogd.8 man page
effectively reverts this commit
988aad01b2 sysklogd: don't use update-alternatives

Signed-off-by: Victor Kamensky 
---
Hi Guys,

Here is more details. Example of failure that I observe:

update-alternatives: Error: not linking 
/home/wd8/oe/20181021/build/tmp-glibc/work/intel_corei7_64-oe-linux/kdevel-console-devel-image/1.0-r0/rootfs/sbin/klogd
 to /usr/lib/busybox/sbin/klogd since 
/home/wd8/oe/20181021/build/tmp-glibc/work/intel_corei7_64-oe-linux/kdevel-console-devel-image/1.0-r0/rootfs/sbin/klogd
 exists and is not a link

Also 988aad01b2 says:

> Using update-alternatives for managing init scripts has proved to be
> problematic. And, sysklogd rconflicts with other syslog daemons so there
> is no point in using update-alternatives from this perspective, either.

I am not sure why "managing init scripts has proved to be problematic" and
syslogd and klogd are not really init script per se, aren't they? Also
klogd and syslogd actually come from busybox, not busybox-syslog as listed
in sysklogd RCONFLICTS. Maybe it what has changed since 988aad01b2.
busybox-syslog now contains only init script for syslog and its
configuration.

Adding Markus for further comments.

 meta/recipes-extended/sysklogd/sysklogd.inc | 5 +
 1 file changed, 5 insertions(+)

diff --git a/meta/recipes-extended/sysklogd/sysklogd.inc 
b/meta/recipes-extended/sysklogd/sysklogd.inc
index f151dd87f7..4393a39180 100644
--- a/meta/recipes-extended/sysklogd/sysklogd.inc
+++ b/meta/recipes-extended/sysklogd/sysklogd.inc
@@ -60,9 +60,14 @@ FILES_${PN} += 
"${@bb.utils.contains('DISTRO_FEATURES','systemd','${exec_prefix}
 
 ALTERNATIVE_PRIORITY = "100"
 
+ALTERNATIVE_${PN} = "syslogd klogd"
+
 ALTERNATIVE_${PN}-doc = "syslogd.8"
 ALTERNATIVE_LINK_NAME[syslogd.8] = "${mandir}/man8/syslogd.8"
 
+ALTERNATIVE_LINK_NAME[syslogd] = "${base_sbindir}/syslogd"
+ALTERNATIVE_LINK_NAME[klogd] = "${base_sbindir}/klogd"
+
 pkg_prerm_${PN} () {
if test "x$D" = "x"; then
if test "$1" = "upgrade" -o "$1" = "remove"; then
-- 
2.17.2

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH] systemtap: 3.3 -> 4.0

[Victor Kamensky via Openembedded-core]

Hi Guys,

On Wed, 17 Oct 2018, Victor Kamensky wrote:


stap-exporter service is a new feature in SystemTap 4.0, I've never used
it before. I'll dig in and will try to make sure that it is functional
in resulting OE image.


I've had a chance to look at stap-exporter and its service more closely.

It looks like it is simple HTTP server that allows to run stap scripts
on target and relay its information to Prometheus monitoring system.
Please check.

https://www.mankier.com/8/stap-exporter

https://prometheus.io

Since it is HTTP server, and its man page says that it is not supposed
to run in untrusted environment I am quite hesitant to have it enabled
by default. Moreover, it comes with some default example scripts that
actually fail to compile in OE core-image-lsb-sdk because it does not
include kernel-vmlinux package by default. stap-exporter is not
prepared to handle script compilation failure and tries to compile it
in the loop, so in core-image-lsb-sdk systemd image where 
stap-exporter.service would be enabled by default it would trash the

system.

Considering all that I suggest stap-exporter to be moved in separate
systemtap-exporter package, so it can be installed explcitely. I
posted patch for it at

http://lists.openembedded.org/pipermail/openembedded-core/2018-October/157073.html

BTW in FC it is separate systemtap-exporter package.

I've tested systemtap-exporter and stap-exporter, if installed
along with proper prerequisites (kernel-vmlinux) is functioning
OK with given default example as far as my manual http testing
with curl shows. I did not test it with prometheus.

Thanks,
Victor
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH] systemtap: move systemtap-exporter into separate package

[Victor Kamensky via Openembedded-core]

stap-exporter runs a set of systemtap scripts and relays
their procfs outputs to remote HTTP clients on demand.

systemtap-exporter is not supposed to run in untrusted
environment. It starts HTTP server on some port. It does
not look safe enough to be included by default along with
the rest of systemtap.

Move systemtap-exporter, its systemd unit, configuration
files and examples scripts into separate package. So if one
needs it and understand its implication, he/she can include
it explicitely.

Signed-off-by: Victor Kamensky 
---
 meta/recipes-kernel/systemtap/systemtap_git.bb | 11 ++-
 1 file changed, 10 insertions(+), 1 deletion(-)

diff --git a/meta/recipes-kernel/systemtap/systemtap_git.bb 
b/meta/recipes-kernel/systemtap/systemtap_git.bb
index 904ecdd106..6ee3e1c0f7 100644
--- a/meta/recipes-kernel/systemtap/systemtap_git.bb
+++ b/meta/recipes-kernel/systemtap/systemtap_git.bb
@@ -27,7 +27,16 @@ PACKAGECONFIG[python3-probes] = 
"--with-python3-probes,--without-python3-probes,
 
 inherit autotools gettext pkgconfig distutils3-base systemd
 
-SYSTEMD_SERVICE_${PN} = "stap-exporter.service"
+PACKAGES =+ "${PN}-exporter"
+
+FILES_${PN}-exporter = "${sysconfdir}/stap-exporter/* \
+${sysconfdir}/sysconfig/stap-exporter \
+${systemd_unitdir}/system/stap-exporter.service \
+${sbindir}/stap-exporter"
+
+RDEPENDS_${PN}-exporter = "${PN} python3-core python3-netclient"
+
+SYSTEMD_SERVICE_${PN}-exporter = "stap-exporter.service"
 
 do_configure_prepend () {
 # Improve reproducibility for c++ object files
-- 
2.17.2

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH] kernel-devsrc: add selinux include files needed by scripts/selinux build

[Victor Kamensky via Openembedded-core]

If CONFIG_SECURITY_SELINUX=y is enabled in kernel configuration, then
'make scripts' command in /usr/src/kernel fails to build
utilities under scripts/selinux that would be pulled in by this config:

  HOSTCC  scripts/selinux/genheaders/genheaders
scripts/selinux/genheaders/genheaders.c:19:10: fatal error: classmap.h: No such 
file or directory
 #include "classmap.h"

To address this issue add security/selinux/include files into
kernel-devsrc.

Signed-off-by: Victor Kamensky 
---
To reproduce this issue add to conf/local.conf selinux fragment:
KERNEL_EXTRA_FEATURES_append = " cgl/features/selinux/selinux.scc"
build core-image-lsb-sdk and run 'cd /usr/src/kernel; make scripts'

 meta/recipes-kernel/linux/kernel-devsrc.bb | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/meta/recipes-kernel/linux/kernel-devsrc.bb 
b/meta/recipes-kernel/linux/kernel-devsrc.bb
index 5758572221..361ad21e1f 100644
--- a/meta/recipes-kernel/linux/kernel-devsrc.bb
+++ b/meta/recipes-kernel/linux/kernel-devsrc.bb
@@ -210,6 +210,9 @@ do_install() {
cp -a --parents kernel/bounds.c $kerneldir/build
cp -a --parents Kbuild $kerneldir/build
fi
+
+# required to build scripts/selinux/genheaders/genheaders
+cp -a --parents security/selinux/include/* $kerneldir/build/
 )
 
 # Make sure the Makefile and version.h have a matching timestamp so that
-- 
2.17.2

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH] systemtap: 3.3 -> 4.0

[Victor Kamensky via Openembedded-core]

On Wed, 17 Oct 2018, Martin Jansa wrote:


systemd.bbclass won't include that .service file in FILES_ without setting 
SYSTEMD_SERVICE_.
I'll send fix for that in few minutes.


Martin, Richard, thank you for the help. I've reproduced
the issue reported by Martin in systemd enabled image, and
verified that his fix solves the issue.

The lesson I learnt:

o Look for WARNINGs

o Test systemd enabled image, along with default sysvinit

stap-exporter service is a new feature in SystemTap 4.0, I've never used
it before. I'll dig in and will try to make sure that it is functional
in resulting OE image.

Thanks,
Victor


On Wed, Oct 17, 2018 at 5:07 PM  wrote:
  On Wed, 2018-10-17 at 16:50 +0200, Martin Jansa wrote:
  > There seems to be one more issue in current master (for which I
  > haven't seen a fix in master-next or ML):
  >
  > ERROR: systemtap-4.0-r0 do_package: QA Issue: systemtap:
  > Files/directories were installed but not shipped in any package:
  >   /lib
  >   /lib/systemd
  >   /lib/systemd/system
  >   /lib/systemd/system/stap-exporter.service
  > Please set FILES such that these items are packaged. Alternatively if
  > they are unneeded, avoid installing them or delete them within
  > do_install.
  > systemtap: 4 installed and not shipped files. [installed-vs-shipped]

  
http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=3b77e7b7852549dcfbc426d4ce258e6e857c0acd
  was supposed to fix that :(

  Cheers,

  Richard


-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH] systemtap: 3.3 -> 4.0

[Victor Kamensky via Openembedded-core]

On Tue, 16 Oct 2018, Richard Purdie wrote:


On Tue, 2018-10-16 at 10:27 +0100, Burton, Ross wrote:

On Mon, 15 Oct 2018 at 19:22, Victor Kamensky 
wrote:

Upgrade systemtap from 3.3 to 4.0: Removed backported patch.


WARNING: systemtap-4.0-r0 do_package_qa: QA Issue: systemtap:
/systemtap/etc/stap-exporter/EXAMPLE is owned by uid 1000, which is
the same as the user running bitbake. This may be due to host
contamination
systemtap: /systemtap/etc/stap-exporter/EXAMPLE_NOERROR is owned by
uid 1000, which is the same as the user running bitbake. This may be
due to host contamination
systemtap: /systemtap/etc/stap-exporter/errno.stp is owned by uid
1000, which is the same as the user running bitbake. This may be due
to host contamination
systemtap: /systemtap/etc/stap-exporter/syscalllatency.stp is owned
by
uid 1000, which is the same as the user running bitbake. This may be
due to host contamination
systemtap: /systemtap/etc/stap-exporter/timeout.stp is owned by uid
1000, which is the same as the user running bitbake. This may be due
to host contamination
systemtap: /systemtap/etc/stap-exporter/topsys.stp is owned by uid
1000, which is the same as the user running bitbake. This may be due
to host contamination [host-user-contaminated]


There are also other issues from the addition of systemd unit files in
4.0. I've a patch in -next which tweaks both issues which I'll put in
for testing.


Richard, thank you very prompt help.

In the long term it would be beneficial that those issues be
fixed in SystemTap itself, correct? If so, I can look at creating
proper patches and pushing them into SystemTap.

Also on my todo list is to attempt to push few existing
OE systemtap patches into SystemTap upstream source. I believe
they look like quite legit issues and I hope they could be
upstreamed.

Thanks,
Victor


Cheers,

Richard



--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH] systemtap: 3.3 -> 4.0

[Victor Kamensky via Openembedded-core]

Hi Randy,

On Mon, 15 Oct 2018, Randy MacLeod wrote:


On 10/15/2018 02:21 PM, Victor Kamensky wrote:

Upgrade systemtap from 3.3 to 4.0: Removed backported patch.

Very short summary of major changes from SystemTap 4.0
announcement by Frank Ch. Eigler :


prometheus exporter network service; ebpf support extensions including
strings and implementation of traditional log(), sprintf() functions;
rebuilt rich tapset coverage for 4.17+ syscalls and for
tracepoint-based syscalls; script language tweaks for supporting
machine-generated scripts


Fixes [YOCTO #12950]

Signed-off-by: Victor Kamensky 
---
Hi Guys,

I'll let you to decide whether you want to accept
it for 2.6 at this point or push it for the next release.
On one hand after move to 4.18 kernel SystemTap 3.3
fails to trace any system call on x86_64 - it is major
functionality breakage. Look at:

https://bugzilla.redhat.com/show_bug.cgi?id=1596456


Ouch!



On other hand it is quite late to do major version
updates.


It is but nothing depends on systemtap so I think we
don't have much choice and we should do it unless there
are autobuilder tests that regress.


Personally, I agree. I had similar thoughts, since it
is leaf project risk is quite contained.



A quick read through the YP mega-manual suggests that
we don't have to change any of the text.
Victor,
Could you take a look and confirm that please?

I noticed that the link in section:
  20.3.3. Documentation
for:
  http://sourceware.org/systemtap/langref/
is dead. There is a pdf:
  http://sourceware.org/systemtap/langref.pdf
Perhaps this is temporary due to the recent release.


Yes, I agree. It is better to change link to langref.pdf
as you suggested. But I don't know how to do that. Is there
somewhere git that holds YP mega-manual sources?



I did go through limited set of tests on all
supported QEMU targets. As far as these tests
concerned proposed 4.0 matches 3.3 results.

Thanks,
Victor


Thanks Victor.

Talk about hot off the presses, 4.0 was released at:
  13-Oct-2018 18:54


Yes :), I was looking how to fix [YOCTO #12950] and was
about to write email to SystemTap guys asking when the
next release is planned and got email from Frank on
SystemTap mailing list announcing 4.0 release. It was
very good timing.

My understanding is that FC28 and FC27 will get SystemTap
4.0 very soon since they moved to 4.18 kernel too.

Thanks,
Victor


$ git log --oneline release-3.3..release-4.0 | wc -l
252

$ git diff release-3.3..release-4.0 | diffstat | tail -1
1574 files changed, 50756 insertions(+), 23604 deletions(-)

Lots of changes to docs, testsuite but also quite a bit to core code
and tapsets:

$ git diff release-3.3..release-4.0 | diffstat  | cut -d"/" -f-2 | uniq -c 
|grep -v "   [1-5] "

   770  b/doc
16  b/httpd
 8  b/man
 9  b/po
18  b/runtime
15  b/stap-exporter
 6  b/staprun
   440  b/tapset
   223  b/testsuite


../Randy



  ...entrypc-avoid-usage-of-uninitialized.patch | 46 ---
  .../systemtap/systemtap_git.inc   |  5 +-
  2 files changed, 2 insertions(+), 49 deletions(-)
  delete mode 100644 
meta/recipes-kernel/systemtap/systemtap/0001-dwflpp-function_entrypc-avoid-usage-of-uninitialized.patch


diff --git 
a/meta/recipes-kernel/systemtap/systemtap/0001-dwflpp-function_entrypc-avoid-usage-of-uninitialized.patch 
b/meta/recipes-kernel/systemtap/systemtap/0001-dwflpp-function_entrypc-avoid-usage-of-uninitialized.patch

deleted file mode 100644
index d0082a1094..00
--- 
a/meta/recipes-kernel/systemtap/systemtap/0001-dwflpp-function_entrypc-avoid-usage-of-uninitialized.patch

+++ /dev/null
@@ -1,46 +0,0 @@
-From 8466fca2a074323a235ef38d425f994a2ff7e64f Mon Sep 17 00:00:00 2001
-From: Victor Kamensky 
-Date: Mon, 9 Jul 2018 09:31:19 -0700
-Subject: [PATCH] dwflpp::function_entrypc avoid usage of uninitialized 
memory

-
-Failure on 3.3 release was observed. Failure was elusive and
-disappeared after seemingly random configure option change, or when
-code was compiled with -O1 or -O0 (vs default -O2). Running failing
-test case under valgrind memcheck pointed to couple places where
-'Conditional jump or move depends on uninitialised value(s)' occured.
-
-After addressing these in two places in dwflpp::function_entrypc,
-valgrind memcheck run is clean and original issue got fixed.
-
-Upstream-Status: Backport
-Signed-off-by: Victor Kamensky 

- dwflpp.cxx | 6 +-
- 1 file changed, 5 insertions(+), 1 deletion(-)
-
-diff --git a/dwflpp.cxx b/dwflpp.cxx
-index bfbb6b096..2172e705a 100644
 a/dwflpp.cxx
-+++ b/dwflpp.cxx
-@@ -2465,13 +2465,17 @@ bool
- dwflpp::function_entrypc (Dwarf_Addr * addr)
- {
-   assert (function);
-+
-+  // assign default value
-+  *addr = 0;
-+
-   // PR10574: reject 0, which tends to be eliminated COMDAT
-   if (dwarf_entrypc (function, addr) == 0 && *addr != 0)
- return true;
-
-   /* Assume the entry pc is the base address, or (if zero)
-  the first ad

[OE-core] [PATCH] systemtap: 3.3 -> 4.0

[Victor Kamensky via Openembedded-core]

Upgrade systemtap from 3.3 to 4.0: Removed backported patch.

Very short summary of major changes from SystemTap 4.0
announcement by Frank Ch. Eigler :

> prometheus exporter network service; ebpf support extensions including
> strings and implementation of traditional log(), sprintf() functions;
> rebuilt rich tapset coverage for 4.17+ syscalls and for
> tracepoint-based syscalls; script language tweaks for supporting
> machine-generated scripts

Fixes [YOCTO #12950]

Signed-off-by: Victor Kamensky 
---
Hi Guys,

I'll let you to decide whether you want to accept
it for 2.6 at this point or push it for the next release.
On one hand after move to 4.18 kernel SystemTap 3.3
fails to trace any system call on x86_64 - it is major
functionality breakage. Look at:

https://bugzilla.redhat.com/show_bug.cgi?id=1596456

On other hand it is quite late to do major version
updates.

I did go through limitted set of tests on all
supported QEMU targets. As far as these tests
concerned proposed 4.0 matches 3.3 results.

Thanks,
Victor

 ...entrypc-avoid-usage-of-uninitialized.patch | 46 ---
 .../systemtap/systemtap_git.inc   |  5 +-
 2 files changed, 2 insertions(+), 49 deletions(-)
 delete mode 100644 
meta/recipes-kernel/systemtap/systemtap/0001-dwflpp-function_entrypc-avoid-usage-of-uninitialized.patch

diff --git 
a/meta/recipes-kernel/systemtap/systemtap/0001-dwflpp-function_entrypc-avoid-usage-of-uninitialized.patch
 
b/meta/recipes-kernel/systemtap/systemtap/0001-dwflpp-function_entrypc-avoid-usage-of-uninitialized.patch
deleted file mode 100644
index d0082a1094..00
--- 
a/meta/recipes-kernel/systemtap/systemtap/0001-dwflpp-function_entrypc-avoid-usage-of-uninitialized.patch
+++ /dev/null
@@ -1,46 +0,0 @@
-From 8466fca2a074323a235ef38d425f994a2ff7e64f Mon Sep 17 00:00:00 2001
-From: Victor Kamensky 
-Date: Mon, 9 Jul 2018 09:31:19 -0700
-Subject: [PATCH] dwflpp::function_entrypc avoid usage of uninitialized memory
-
-Failure on 3.3 release was observed. Failure was elusive and
-disappeared after seemingly random configure option change, or when
-code was compiled with -O1 or -O0 (vs default -O2). Running failing
-test case under valgrind memcheck pointed to couple places where
-'Conditional jump or move depends on uninitialised value(s)' occured.
-
-After addressing these in two places in dwflpp::function_entrypc,
-valgrind memcheck run is clean and original issue got fixed.
-
-Upstream-Status: Backport
-Signed-off-by: Victor Kamensky 

- dwflpp.cxx | 6 +-
- 1 file changed, 5 insertions(+), 1 deletion(-)
-
-diff --git a/dwflpp.cxx b/dwflpp.cxx
-index bfbb6b096..2172e705a 100644
 a/dwflpp.cxx
-+++ b/dwflpp.cxx
-@@ -2465,13 +2465,17 @@ bool
- dwflpp::function_entrypc (Dwarf_Addr * addr)
- {
-   assert (function);
-+
-+  // assign default value
-+  *addr = 0;
-+
-   // PR10574: reject 0, which tends to be eliminated COMDAT
-   if (dwarf_entrypc (function, addr) == 0 && *addr != 0)
- return true;
- 
-   /* Assume the entry pc is the base address, or (if zero)
-  the first address of the ranges covering this DIE.  */
--  Dwarf_Addr start, end;
-+  Dwarf_Addr start = 0, end;
-   if (dwarf_ranges (function, 0, addr, , ) >= 0)
- {
-   if (*addr == 0)
--- 
-2.17.1
-
diff --git a/meta/recipes-kernel/systemtap/systemtap_git.inc 
b/meta/recipes-kernel/systemtap/systemtap_git.inc
index 06924fc240..274fcde5c1 100644
--- a/meta/recipes-kernel/systemtap/systemtap_git.inc
+++ b/meta/recipes-kernel/systemtap/systemtap_git.inc
@@ -1,7 +1,7 @@
 LICENSE = "GPLv2"
 LIC_FILES_CHKSUM = "file://COPYING;md5=b234ee4d69f5fce4486a80fdaf4a4263"
-SRCREV = "48867d1cface9445d58e3c6e14497770b7eb77e1"
-PV = "3.3"
+SRCREV = "428f84e9e656bce71018e8902e4edb8aacafcc0e"
+PV = "4.0"
 
 SRC_URI = "git://sourceware.org/git/systemtap.git \
file://configure-allow-to-disable-libvirt.patch \
@@ -11,7 +11,6 @@ SRC_URI = "git://sourceware.org/git/systemtap.git \

file://0001-Do-not-let-configure-write-a-python-location-into-th.patch \
file://0001-Install-python-modules-to-correct-library-dir.patch \

file://0001-staprun-stapbpf-don-t-support-installing-a-non-root.patch \
-   
file://0001-dwflpp-function_entrypc-avoid-usage-of-uninitialized.patch \
"
 
 COMPATIBLE_HOST = '(x86_64|i.86|powerpc|arm|aarch64|microblazeel|mips).*-linux'
-- 
2.17.1

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 1/2] cve-report: add scripts to generate CVE reports

[Victor Kamensky via Openembedded-core]

On Sat, 4 Aug 2018, Alexander Kanavin wrote:


How reliable is NVD database for such automated scans? Previously, we
have repeatedly concluded that it should not be trusted, and proper
patching of vulnerabilities must involve humans looking at
vulnerability reports and making appropriate decisions - same as
Debian is doing for example.


I don't think scanning NVD database and generating proper
reports and having people looking at vulnerability reports are
mutually exclusive. I think rather both should be done.

The patches and tools we are proposing do scanning and
reporting, while looking at versions, pattches applied, for
a given image. Correctly analazying CVEs and applying right
patches is a separate thing.

Just an example please find below reports generated for sumo
that I pulled today, core-image-lsb image qemux86-64 MACHINE.

As one may see in below report it is not perfect. There are
clearly false positives, misplaced entries based on, as you
mentioned on not fully reliable NVD data. I.e CVE-2013-4598
reported against gcc the one I could spot - it is wrong. But it
does not negate value of cases where it is reported correctly.
So it boils down to a question whether value of useful signal
against cost of filtering some noise. From our experience using
this tool for a while internally to fix outstanding CVEs for
krogoth branch, the right signal definitely has very good value.

Grygorii, btw I had trouble pulling your patches from email,
it seems that there were posted as Text/HTML (at least it is what
my mailer tells me). Could you please repost them as Text/PLAIN,
i.e using git send-mail

report-foss.txt:

  patched |  5.5 | CVE-2017-15873 | busybox | 1.27.2
  patched |  8.8 | CVE-2017-16544 | busybox | 1.27.2
  patched |  6.5 | CVE-2016-3189  | bzip2 | 1.0.6
  patched |  7.5 | CVE-2017-9814  | cairo | 1.14.12
  patched |  9.8 | CVE-2016-6354  | flex | 2.6.0
  patched |  7.3 | CVE-2015-8560  | foomatic-filters | 4.0.17
  patched |  7.5 | CVE-2015-8327  | foomatic-filters | 4.0.17
  patched |  7.8 | CVE-2017-11714 | ghostscript | 9.21
  patched |  7.8 | CVE-2017-9835  | ghostscript | 9.21
  patched |  8.1 | CVE-2018-11236 | glibc | 2.27
  patched |  6.5 | CVE-2017-14166 | libarchive | 3.3.2
  patched |  7.5 | CVE-2017-14502 | libarchive | 3.3.2
  patched |  5.5 | CVE-2017-8361  | libsndfile | 1.0.28
  patched |  5.5 | CVE-2017-8362  | libsndfile | 1.0.28
  patched |  5.5 | CVE-2017-8363  | libsndfile | 1.0.28
  patched |  5.5 | CVE-2017-8365  | libsndfile | 1.0.28
  patched |  8.8 | CVE-2017-6892  | libsndfile | 1.0.28
  patched |  6.5 | CVE-2017-18013 | libtiff | 4.0.9
  patched |  6.5 | CVE-2018-10963 | libtiff | 4.0.9
  patched |  6.5 | CVE-2018-5784  | libtiff | 4.0.9
  patched |  6.5 | CVE-2018-7456  | libtiff | 4.0.9
  patched |  8.8 | CVE-2018-8905  | libtiff | 4.0.9
  patched |  6.5 | CVE-2017-14633 | libvorbis | 1.3.5
  patched |  9.8 | CVE-2017-14632 | libvorbis | 1.3.5
  patched |  7.5 | CVE-2018-6951  | patch | 2.7.6
  patched |  7.8 | CVE-2018-1000156   | patch | 2.7.6
  patched |  7.5 | CVE-2017-12837 | perl | 5.24.1
  patched |  9.1 | CVE-2017-12883 | perl | 5.24.1
  patched |  7.5 | CVE-2017-8779  | rpcbind | 0.2.4
  patched |  4.0 | CVE-2014-9913  | unzip | 6.0
  patched |  4.0 | CVE-2016-9844  | unzip | 6.0
  patched |  4.3 | CVE-2015-7697  | unzip | 6.0
  patched |  5.0 | CVE-2014-9636  | unzip | 6.0
  patched |  6.8 | CVE-2015-7696  | unzip | 6.0
  patched |  5.3 | CVE-2017-13078 | wpa_supplicant | 2.6
  patched |  5.3 | CVE-2017-13079 | wpa_supplicant | 2.6
  patched |  5.3 | CVE-2017-13080 | wpa_supplicant | 2.6
  patched |  5.3 | CVE-2017-13081 | wpa_supplicant | 2.6
  patched |  5.3 | CVE-2017-13087 | wpa_supplicant | 2.6
  patched |  5.3 | CVE-2017-13088 | wpa_supplicant | 2.6
  patched |  6.8 | CVE-2017-13077 | wpa_supplicant | 2.6
  patched |  6.8 | CVE-2017-13086 | wpa_supplicant | 2.6
  patched |  8.1 | CVE-2017-13082 | wpa_supplicant | 2.6
unpatched |  5.5 | CVE-2018-10372 | binutils | 2.30
unpatched |  5.5 | CVE-2018-10534 | binutils | 2.30
unpatched |  5.5 | CVE-2018-10535 | binutils | 2.30
unpatched |  5.5 | CVE-2018-6759  | binutils | 2.30
unpatched |  5.5 | CVE-2018-6872  | binutils | 2.30
unpatched |  5.5 | CVE-2018-7568  | binutils | 2.30
unpatched |  5.5 | CVE-2018-7569  | binutils | 2.30
unpatched |  5.5 | CVE-2018-7570  | binutils | 2.30
unpatched |  5.5 | CVE-2018-7642  | binutils | 2.30
unpatched |  5.5 | CVE-2018-8945  | binutils | 2.30
unpatched |  5.5 | CVE-2018-9138  | binutils | 2.30
unpatched |  5.5 | CVE-2018-9996  | binutils | 2.30
unpatched |  6.5 | CVE-2018-10373 | binutils | 2.30
unpatched |  7.8 | CVE-2018-6543  | binutils | 2.30
unpatched |  7.8 | CVE-2018-7208  | binutils | 2.30
unpatched |  7.8 | CVE-2018-7643  

Re: [OE-core] [PATCH] package: skip strip on signed kernel modules

[Victor Kamensky via Openembedded-core]

On Fri, 3 Aug 2018, Ocampo Coronado, Omar wrote:


Yes, we would like to keep the symbols on a signed kernel module.

Andre shared this link:  
https://www.kernel.org/doc/html/v4.17/admin-guide/module-signing.html#signed-modules-and-stripping
 , from conversation topic: Re: [OE-core] Strip kernel modules and signatures


Thank you for the pointer. I did not expect that KLM signing
will be outside of ELF. Too bad that it is so brittle.

Ideally, it would be nice if one could disable KLM signing in
kernel makefile machinery and have mechanism to sign KLMs in OE
itself, just before packaging but after they got stripped.
IMO it would be more practical. I could not imagine if one
would want to ship KLMs with debug symbols inside. But even
if that is implemented, your code would still should stand ok -
if module signed already, it cannot be touched.


-28 are the last 28 bytes of the file. The same amount of bytes are being read 
by dracut to check if a module is signed.
And you are correct Victor, I'm unsure if this would work outside x86 arch.


I've checked that by building mips64 kernel with KLM signing
enabled and I looked at scripts/sign-file.c source, you are
fine: magic_number = "~Module signature appended~\n" will be
always at the end after KLM signing regardless of architecture.

Thanks,
Victor


Two pending fixes:
   1) This patch also needs to fix the mode of the file as the original may not 
be preserved.
   2)  Seems like 'return' is not accepted by oe.utils.multiprocess, still 
getting familiar with OE

-Original Message-----
From: Victor Kamensky [mailto:kamen...@cisco.com]
Sent: Friday, August 3, 2018 3:28 PM
To: Ocampo Coronado, Omar 
Cc: openembedded-core@lists.openembedded.org
Subject: Re: [OE-core] [PATCH] package: skip strip on signed kernel modules



On Fri, 3 Aug 2018, omar.ocampo.coron...@intel.com wrote:


From: foocampo 

Executing strip action on kernel modules removes the signature.
Is not possible to strip and keep the signature, therefore avoid strip
signed kernel modules.

Signed-off-by: foocampo 
---
meta/lib/oe/package.py | 10 ++
1 file changed, 10 insertions(+)

diff --git a/meta/lib/oe/package.py b/meta/lib/oe/package.py index
fa3428ad61..f7d2d3b7c4 100644
--- a/meta/lib/oe/package.py
+++ b/meta/lib/oe/package.py
@@ -24,6 +24,9 @@ def runstrip(arg):

# kernel module
if elftype & 16:
+if is_kernel_module_signed(file):
+bb.debug(1, "Skip strip on signed module %s" % file)
+return


It does not look right to me. Above means that signed KLM will go into image 
with symbols. Or I don't read code correctly?

Where is signature stored? Is it some kind of an ELF NOTE? In this case you would just 
need to drop only "--remove-section=.note"
from strip command. Wondering why .notes were stripped in the first place.


stripcmd.extend(["--strip-debug", "--remove-section=.comment",
"--remove-section=.note", "--preserve-dates"])


I suggest split above into two invocations and do second
stripcmd.extend(["--remove-section=.note"]) only for non signed modules.
Assuming that signature is in the .note section. If it is not .comment, do that with 
"--remove-section=.comment" instead.


# .so and shared library
@@ -46,6 +49,13 @@ def is_kernel_module(path):
with open(path) as f:
return mmap.mmap(f.fileno(), 0,
prot=mmap.PROT_READ).find(b"vermagic=") >= 0

+# Detect if .ko module is signed
+def is_kernel_module_signed(path):
+with open(path, "rb") as f:
+f.seek(-28, 2)


Where magic -28 comes from? Is it true for all cases, all CPU arches?
I think it could be done more cleanly here.

Thanks,
Victor


+module_tail = f.read()
+return "Module signature appended" in "".join(chr(c) for c in
+ bytearray(module_tail))
+
# Return type (bits):
# 0 - not elf
# 1 - ELF
--
2.18.0

--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core




--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH] package: skip strip on signed kernel modules

[Victor Kamensky via Openembedded-core]

On Fri, 3 Aug 2018, omar.ocampo.coron...@intel.com wrote:


From: foocampo 

Executing strip action on kernel modules removes the signature.
Is not possible to strip and keep the signature, therefore avoid
strip signed kernel modules.

Signed-off-by: foocampo 
---
meta/lib/oe/package.py | 10 ++
1 file changed, 10 insertions(+)

diff --git a/meta/lib/oe/package.py b/meta/lib/oe/package.py
index fa3428ad61..f7d2d3b7c4 100644
--- a/meta/lib/oe/package.py
+++ b/meta/lib/oe/package.py
@@ -24,6 +24,9 @@ def runstrip(arg):

# kernel module
if elftype & 16:
+if is_kernel_module_signed(file):
+bb.debug(1, "Skip strip on signed module %s" % file)
+return


It does not look right to me. Above means that signed
KLM will go into image with symbols. Or I don't read
code correctly?

Where is signature stored? Is it some kind of an ELF NOTE? In
this case you would just need to drop only "--remove-section=.note"
from strip command. Wondering why .notes were stripped in the
first place.


stripcmd.extend(["--strip-debug", "--remove-section=.comment",
"--remove-section=.note", "--preserve-dates"])


I suggest split above into two invocations and do second
stripcmd.extend(["--remove-section=.note"]) only for non signed modules.
Assuming that signature is in the .note section. If it is not .comment,
do that with "--remove-section=.comment" instead.


# .so and shared library
@@ -46,6 +49,13 @@ def is_kernel_module(path):
with open(path) as f:
return mmap.mmap(f.fileno(), 0, prot=mmap.PROT_READ).find(b"vermagic=") 
>= 0

+# Detect if .ko module is signed
+def is_kernel_module_signed(path):
+with open(path, "rb") as f:
+f.seek(-28, 2)


Where magic -28 comes from? Is it true for all cases, all CPU arches?
I think it could be done more cleanly here.

Thanks,
Victor


+module_tail = f.read()
+return "Module signature appended" in "".join(chr(c) for c in 
bytearray(module_tail))
+
# Return type (bits):
# 0 - not elf
# 1 - ELF
--
2.18.0

--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core


--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH 2/2] systemtap: fix unintialized memory accesses in dwflpp::function_entrypc

[Victor Kamensky via Openembedded-core]

Observed failure in SystemTap v3.3 unit testing, It was tracked down
to unintialized memory access in dwflpp::function_entrypc method.

Upstream-Status: Backport
Signed-off-by: Victor Kamensky 
---
 ...entrypc-avoid-usage-of-uninitialized.patch | 46 +++
 .../systemtap/systemtap_git.inc   |  1 +
 2 files changed, 47 insertions(+)
 create mode 100644 
meta/recipes-kernel/systemtap/systemtap/0001-dwflpp-function_entrypc-avoid-usage-of-uninitialized.patch

diff --git 
a/meta/recipes-kernel/systemtap/systemtap/0001-dwflpp-function_entrypc-avoid-usage-of-uninitialized.patch
 
b/meta/recipes-kernel/systemtap/systemtap/0001-dwflpp-function_entrypc-avoid-usage-of-uninitialized.patch
new file mode 100644
index 00..d0082a1094
--- /dev/null
+++ 
b/meta/recipes-kernel/systemtap/systemtap/0001-dwflpp-function_entrypc-avoid-usage-of-uninitialized.patch
@@ -0,0 +1,46 @@
+From 8466fca2a074323a235ef38d425f994a2ff7e64f Mon Sep 17 00:00:00 2001
+From: Victor Kamensky 
+Date: Mon, 9 Jul 2018 09:31:19 -0700
+Subject: [PATCH] dwflpp::function_entrypc avoid usage of uninitialized memory
+
+Failure on 3.3 release was observed. Failure was elusive and
+disappeared after seemingly random configure option change, or when
+code was compiled with -O1 or -O0 (vs default -O2). Running failing
+test case under valgrind memcheck pointed to couple places where
+'Conditional jump or move depends on uninitialised value(s)' occured.
+
+After addressing these in two places in dwflpp::function_entrypc,
+valgrind memcheck run is clean and original issue got fixed.
+
+Upstream-Status: Backport
+Signed-off-by: Victor Kamensky 
+---
+ dwflpp.cxx | 6 +-
+ 1 file changed, 5 insertions(+), 1 deletion(-)
+
+diff --git a/dwflpp.cxx b/dwflpp.cxx
+index bfbb6b096..2172e705a 100644
+--- a/dwflpp.cxx
 b/dwflpp.cxx
+@@ -2465,13 +2465,17 @@ bool
+ dwflpp::function_entrypc (Dwarf_Addr * addr)
+ {
+   assert (function);
++
++  // assign default value
++  *addr = 0;
++
+   // PR10574: reject 0, which tends to be eliminated COMDAT
+   if (dwarf_entrypc (function, addr) == 0 && *addr != 0)
+ return true;
+ 
+   /* Assume the entry pc is the base address, or (if zero)
+  the first address of the ranges covering this DIE.  */
+-  Dwarf_Addr start, end;
++  Dwarf_Addr start = 0, end;
+   if (dwarf_ranges (function, 0, addr, , ) >= 0)
+ {
+   if (*addr == 0)
+-- 
+2.17.1
+
diff --git a/meta/recipes-kernel/systemtap/systemtap_git.inc 
b/meta/recipes-kernel/systemtap/systemtap_git.inc
index a1e05579e6..06924fc240 100644
--- a/meta/recipes-kernel/systemtap/systemtap_git.inc
+++ b/meta/recipes-kernel/systemtap/systemtap_git.inc
@@ -11,6 +11,7 @@ SRC_URI = "git://sourceware.org/git/systemtap.git \

file://0001-Do-not-let-configure-write-a-python-location-into-th.patch \
file://0001-Install-python-modules-to-correct-library-dir.patch \

file://0001-staprun-stapbpf-don-t-support-installing-a-non-root.patch \
+   
file://0001-dwflpp-function_entrypc-avoid-usage-of-uninitialized.patch \
"
 
 COMPATIBLE_HOST = '(x86_64|i.86|powerpc|arm|aarch64|microblazeel|mips).*-linux'
-- 
2.17.1

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH 1/2] systemtap: 3.2 -> 3.3

[Victor Kamensky via Openembedded-core]

Upgrade systemtap from 3.2 to 3.3: Removed all backported patches.
Removed "remove quotes around -I include" pending patch since 3.3
got similar fix already. Resolved merge conflict in and
regenerated monitor-option.patch patch.

Signed-off-by: Victor Kamensky 
---
 ...dded-a-couple-of-small-sysroot-fixes.patch |  42 ---
 ...root-path-to-module-name-in-case-of-.patch |  61 
 ...pdating-the-use-of-timers-for-the-4..patch | 277 --
 .../systemtap/0001-Fixes-for-gcc-8.patch  | 215 --
 ...sysroot-paths-don-t-end-with-a-slash.patch | 128 
 ...when-looking-for-the-System.map-file.patch |  29 --
 ...ocate-needs-target-file-path-not-hos.patch |  39 ---
 ...-remove-quotes-around-I-include-line.patch |  38 ---
 ...-with-sysroot-case-do-not-remove-sys.patch |  42 ---
 ...t-release-r-option-handling-follow-u.patch |  40 ---
 ...-fix-short-release-r-option-handling.patch |  53 
 ...ymbolic-links-with-absolute-name-rel.patch | 117 
 .../systemtap/systemtap/monitor-option.patch  |  15 +-
 .../systemtap/systemtap_git.inc   |  16 +-
 14 files changed, 8 insertions(+), 1104 deletions(-)
 delete mode 100644 
meta/recipes-kernel/systemtap/systemtap/0001-Added-a-couple-of-small-sysroot-fixes.patch
 delete mode 100644 
meta/recipes-kernel/systemtap/systemtap/0001-Delay-adding-sysroot-path-to-module-name-in-case-of-.patch
 delete mode 100644 
meta/recipes-kernel/systemtap/systemtap/0001-Fix-PR22551-by-updating-the-use-of-timers-for-the-4..patch
 delete mode 100644 
meta/recipes-kernel/systemtap/systemtap/0001-Fixes-for-gcc-8.patch
 delete mode 100644 
meta/recipes-kernel/systemtap/systemtap/0001-Make-sure-sysroot-paths-don-t-end-with-a-slash.patch
 delete mode 100644 
meta/recipes-kernel/systemtap/systemtap/0001-Use-sysroot-when-looking-for-the-System.map-file.patch
 delete mode 100644 
meta/recipes-kernel/systemtap/systemtap/0001-_stp_umodule_relocate-needs-target-file-path-not-hos.patch
 delete mode 100644 
meta/recipes-kernel/systemtap/systemtap/0001-buildrun-remove-quotes-around-I-include-line.patch
 delete mode 100644 
meta/recipes-kernel/systemtap/systemtap/0001-debuginfo-lookup-with-sysroot-case-do-not-remove-sys.patch
 delete mode 100644 
meta/recipes-kernel/systemtap/systemtap/0001-sysroot-fix-short-release-r-option-handling-follow-u.patch
 delete mode 100644 
meta/recipes-kernel/systemtap/systemtap/0001-sysroot-fix-short-release-r-option-handling.patch
 delete mode 100644 
meta/recipes-kernel/systemtap/systemtap/0001-sysroot-handle-symbolic-links-with-absolute-name-rel.patch

diff --git 
a/meta/recipes-kernel/systemtap/systemtap/0001-Added-a-couple-of-small-sysroot-fixes.patch
 
b/meta/recipes-kernel/systemtap/systemtap/0001-Added-a-couple-of-small-sysroot-fixes.patch
deleted file mode 100644
index c0ceb5a412..00
--- 
a/meta/recipes-kernel/systemtap/systemtap/0001-Added-a-couple-of-small-sysroot-fixes.patch
+++ /dev/null
@@ -1,42 +0,0 @@
-From a714658727206d2a98a7194b7e6d29dbd3e27b8d Mon Sep 17 00:00:00 2001
-From: David Smith 
-Date: Mon, 19 Mar 2018 16:50:05 -0500
-Subject: [PATCH] Added a couple of small sysroot fixes.
-
-* tapsets.cxx (dwarf_builder::build): Fix commit 4ffecddf5.
-  (path_remove_sysroot): Fix extra '/' present at start of paths.
-
-Upstream-Status: Backport
-Signed-off-by: Victor Kamensky 

- tapsets.cxx | 10 +++---
- 1 file changed, 7 insertions(+), 3 deletions(-)
-
-Index: git/tapsets.cxx
-===
 git.orig/tapsets.cxx
-+++ git/tapsets.cxx
-@@ -1395,7 +1395,8 @@ string path_remove_sysroot(const systemt
-   string retval = path;
-   if (!sess.sysroot.empty() &&
-   (pos = retval.find(sess.sysroot)) != string::npos)
--retval.replace(pos, sess.sysroot.length(), "/");
-+retval.replace(pos, sess.sysroot.length(),
-+ (sess.sysroot.back() == '/' ? "/": ""));
-   return retval;
- }
- 
-@@ -8412,8 +8413,11 @@ dwarf_builder::build(systemtap_session &
- 
-   // PR13338: unquote glob results
-   module_name = unescape_glob_chars (module_name);
--  user_path = find_executable (module_name, "", sess.sysenv); // 
canonicalize it
--  if (!is_fully_resolved(user_path, sess.sysroot, sess.sysenv))
-+  user_path = find_executable (module_name, sess.sysroot, sess.sysenv); 
// canonicalize it
-+  // Note we don't need to pass the sysroot to
-+  // is_fully_resolved(), since we just passed it to
-+  // find_executable().
-+  if (!is_fully_resolved(user_path, "", sess.sysenv))
- throw SEMANTIC_ERROR(_F("cannot find executable '%s'",
- user_path.to_string().c_str()));
- 
diff --git 
a/meta/recipes-kernel/systemtap/systemtap/0001-Delay-adding-sysroot-path-to-module-name-in-case-of-.patch
 
b/meta/recipes-kernel/systemtap/systemtap/0001-Delay-adding-sysroot-path-to-module-name-in-case-of-.patc

Re: [OE-core] [PATCH v3 5/5] crosstap: replace script with new python based implementation

[Victor Kamensky]

On Mon, 14 May 2018, Burton, Ross wrote:


On 5 April 2018 at 19:25, Victor Kamensky <kamen...@cisco.com> wrote:

New crosstap python implementation is total replacement for
crosstap shell script, that has superseding capabilities.
New script support cross compiling of SystemTap scripts
for user-land, by using supplied image rootfs. Whereas old
script could only deal with scripts against kernel. New script
has more complex logic and additional capabilities.


...


+benv_kernel = BitbakeEnv("virtual/kernel")
+(self.staging_bindir_toolchain,
+ self.target_prefix,
+ self.target_arch,
+ self.target_kernel_builddir
+) = benv_kernel.get_vars(
+ ("STAGING_BINDIR_TOOLCHAIN",
+  "TARGET_PREFIX",
+  "TRANSLATED_TARGET_ARCH",
+  "B"
+))



+if not os.path.isdir(self.target_kernel_builddir):
+print("ERROR: Cannot find '" + self.target_kernel_builddir +
+  "' directory. Was 'kernel/virtual' built?")
+ret = False


You can't assume that B exists and is useful, as if virtual/kernel was
installed from sstate then all the installed pieces will be present
but $B will be empty.


Ross, thank you for pointing to this. I will look at it.

Please note use of B of kernel/virtual by crosstap preceeded my
commit:

c6da6b64 (Tom Zanussi 2012-08-31 00:06:57 -0500 109) 
TARGET_KERNEL_BUILDDIR=$(echo "$BITBAKE_VARS" | grep ^B= \
c6da6b64 (Tom Zanussi 2012-08-31 00:06:57 -0500 110)   | cut -d '=' -f2 | 
cut -d '"' -f2)


I just followed the same logic.


For example, I re-ran oe-selftest on the autobuilder, the kernel was
extracted from sstate as the previous run built it, and the test
failed:


Systemtap translator generates kernel module source and it needs
cross compiled environment to build it against proper kernel sources
build artifacts. I.e cross compilation of kernel module should
be supported when kernel is extracted from sstate.

So my thought was to figure out how to cross compile kernel module
against kernel/virtual sstate, I can take a look how recipe that
'inherit module' works in such case. So I've tried the following in
tree pulled today (cryptodev-module is recipe example that
does 'inherit module'):

Variant1:

bitbake cryptodev-module
bitbake -c clean linux-yocto
bitbake -c cleansstate cryptodev-module
bitbake cryptodev-module

Variant2:

bitbake cryptodev-module
mv tmp-glibc tmp-glibc.bak
bitbake linux-yocto
bitbake cryptodev-module

In both cases I see that full kernel rebuild (pull source, compile,
etc) is retriggered by building cryptodev-module, even if
linux-yocto initially is pulled from sstate. In Variant1
cryptodev-module build actually failed because some host utilities
from kernel tree (like scripts/basic/fixdep) were not rebuilt.

I don't think it is proper for crosstap utility to retrigger
full kernel/virtual build. So unless kernel module build against
kernel/virtual sstate without full rebuild of kernel/virtual is
solved I am not sure what I can do in crosstap script.

Currently I see that work-shared/$MACHINE/kernel-source and
work-shared/$MACHINE/kernel-build-artifacts are created and it
seems their purpose is to facilitate external kernel modules
build, and maybe need to full kernel rebuild will be fixed soon.

Do I miss something? Any advise how to address this problem
is appreciated.

Thanks,
Victor


2018-05-13 22:36:08,905 - oe-selftest - INFO - FAIL:
test_crosstap_can_use_systemtap_on_qemu (crosstap.CrossTapTest)
2018-05-13 22:36:08,906 - oe-selftest - INFO -
--
2018-05-13 22:36:08,906 - oe-selftest - INFO - Traceback (most recent
call last):
 File 
"/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/meta/lib/oeqa/selftest/cases/crosstap.py",
line 25, in test_crosstap_can_use_systemtap_on_qemu
   self.assertIn('hello world', result.output, 'Crosstap failed.')
AssertionError: 'hello world' not found in 'ERROR: Cannot find
\'/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/build/tmp/work/qemux86_64-poky-linux/linux-yocto/4.14.30+gitAUTOINC+ea9330894e_74f6cd2b69-r0/linux-qemux86_64-standard-build\'
directory. Was \'kernel/virtual\' built?

Ross


--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH v3 5/5] crosstap: replace script with new python based implementation

[Victor Kamensky]

New crosstap python implementation is total replacement for
crosstap shell script, that has superseding capabilities.
New script support cross compiling of SystemTap scripts
for user-land, by using supplied image rootfs. Whereas old
script could only deal with scripts against kernel. New script
has more complex logic and additional capabilities.

As invocation interface new script support old "legacy"
mode and provides alternative new regular options interface
to access additional functionality.

Signed-off-by: Victor Kamensky <kamen...@cisco.com>
---
 scripts/crosstap | 586 ++-
 1 file changed, 450 insertions(+), 136 deletions(-)

diff --git a/scripts/crosstap b/scripts/crosstap
index 39739bb..e33fa4a 100755
--- a/scripts/crosstap
+++ b/scripts/crosstap
@@ -1,15 +1,22 @@
-#!/bin/bash
+#!/usr/bin/env python3
 #
-# Run a systemtap script on remote target
+# Build a systemtap script for a given image, kernel
 #
-# Examples (run on build host, target is 192.168.1.xxx):
-#   $ source oe-init-build-env"
-#   $ cd ~/my/systemtap/scripts"
+# Effectively script extracts needed information from set of
+# 'bitbake -e' commands and contructs proper invocation of stap on
+# host to build systemtap script for a given target.
 #
-#   $ crosstap r...@192.168.1.xxx myscript.stp"
-#   $ crosstap r...@192.168.1.xxx myscript-with-args.stp 99 ninetynine"
+# By default script will compile scriptname.ko that could be copied
+# to taget and activated with 'staprun scriptname.ko' command. Or if
+# --remote user@hostname option is specified script will build, load
+# execute script on target.
 #
-# Copyright (c) 2012, Intel Corporation.
+# This script is very similar and inspired by crosstap shell script.
+# The major difference that this script supports user-land related
+# systemtap script, whereas crosstap could deal only with scripts
+# related to kernel.
+#
+# Copyright (c) 2018, Cisco Systems.
 # All rights reserved.
 #
 # This program is free software; you can redistribute it and/or modify
@@ -25,131 +32,438 @@
 # along with this program; if not, write to the Free Software
 # Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
 
-function usage() {
-echo "Usage: $0 <user@hostname>  [additional 
systemtap-script args]"
-}
-
-function setup_usage() {
-echo ""
-echo "'crosstap' requires a local sdk build of the target system"
-echo "(or a build that includes 'tools-profile') in order to build"
-echo "kernel modules that can probe the target system."
-echo ""
-echo "Practically speaking, that means you need to do the following:"
-echo "  - If you're running a pre-built image, download the release"
-echo "and/or BSP tarballs used to build the image."
-echo "  - If you're working from git sources, just clone the metadata"
-echo "and BSP layers needed to build the image you'll be booting."
-echo "  - Make sure you're properly set up to build a new image (see"
-echo "the BSP README and/or the widely available basic documentation"
-echo "that discusses how to build images)."
-echo "  - Build an -sdk version of the image e.g.:"
-echo "  $ bitbake core-image-sato-sdk"
-echo "  OR"
-echo "  - Build a non-sdk image but include the profiling tools:"
-echo "  [ edit local.conf and add 'tools-profile' to the end of"
-echo "the EXTRA_IMAGE_FEATURES variable ]"
-echo "  $ bitbake core-image-sato"
-echo ""
-echo "  [ NOTE that 'crosstap' needs to be able to ssh into the target"
-echo "system, which isn't enabled by default in -minimal images. ]"
-echo ""
-echo "Once you've build the image on the host system, you're ready to"
-echo "boot it (or the equivalent pre-built image) and use 'crosstap'"
-echo "to probe it (you need to source the environment as usual first):"
-echo ""
-echo "$ source oe-init-build-env"
-echo "$ cd ~/my/systemtap/scripts"
-echo "$ crosstap r...@192.168.1.xxx myscript.stp"
-echo ""
-}
-
-function systemtap_target_arch() {
-SYSTEMTAP_TARGET_ARCH=$1
-case $SYSTEMTAP_TARGET_ARCH in
-i?86)
-SYSTEMTAP_TARGET_ARCH="i386"
-;;
-x86?64*)
-SYSTEMTAP_TARGET_ARCH="x86_64"
-;;
-arm*)
-SYSTEMTAP_TARGET_ARCH="arm"
-;;
-powerpc*)
-SYSTEMTAP_TARGET_ARCH="powerpc"
-;;
-*)
-;;
-esac
-}
-
-if [ $# -lt 2 ]; then
-   u

[OE-core] [PATCH v3 4/5] systemtap: create translator packageconfig

[Victor Kamensky]

For cases when systemap module compilation happens on host in
cross-compilation mode, and it is desirable to minimize systemtap
presense on target we need to have just smallest possible set of
utilties that are required to run compiled modules.

Introduce new "translator" PACKAGECONFIG, if it is not set
it would mean that just minimal set of run-time utilities will
be included in the package.

For run-time only systemtap build variant use
PACKAGECONFIG_pn-systemtap = "" or
PACKAGECONFIG_pn-systemtap = "monitor"

Suggested-by: Taras Kondratiuk <takon...@cisco.com>
Signed-off-by: Victor Kamensky <kamen...@cisco.com>
---
 meta/recipes-kernel/systemtap/systemtap_git.bb | 15 +++
 1 file changed, 11 insertions(+), 4 deletions(-)

diff --git a/meta/recipes-kernel/systemtap/systemtap_git.bb 
b/meta/recipes-kernel/systemtap/systemtap_git.bb
index 475b207..b280f58 100644
--- a/meta/recipes-kernel/systemtap/systemtap_git.bb
+++ b/meta/recipes-kernel/systemtap/systemtap_git.bb
@@ -3,9 +3,7 @@ HOMEPAGE = "https://sourceware.org/systemtap/;
 
 require systemtap_git.inc
 
-DEPENDS = "boost elfutils"
-
-RDEPENDS_${PN} += "python3-core bash perl"
+DEPENDS = "elfutils"
 
 EXTRA_OECONF += "--with-libelf=${STAGING_DIR_TARGET} --without-rpm \
 --without-nss --without-avahi --without-dyninst \
@@ -18,7 +16,8 @@ STAP_DOCS ?= "--disable-docs --disable-publican 
--disable-refdocs"
 
 EXTRA_OECONF += "${STAP_DOCS} "
 
-PACKAGECONFIG ??= "sqlite monitor python3-probes"
+PACKAGECONFIG ??= "translator sqlite monitor python3-probes"
+PACKAGECONFIG[translator] = 
"--enable-translator,--disable-translator,boost,python3-core bash perl"
 PACKAGECONFIG[libvirt] = "--enable-libvirt,--disable-libvirt,libvirt"
 PACKAGECONFIG[sqlite] = "--enable-sqlite,--disable-sqlite,sqlite3"
 PACKAGECONFIG[monitor] = "--enable-monitor,--disable-monitor,ncurses json-c"
@@ -26,4 +25,12 @@ PACKAGECONFIG[python3-probes] = 
"--with-python3-probes,--without-python3-probes,
 
 inherit autotools gettext pkgconfig distutils3-base
 
+do_install_append () {
+   if [ ! -f ${D}${bindir}/stap ]; then
+  # translator disabled case, need to leave only minimal runtime
+  rm -rf ${D}${datadir}/${PN}
+  rm ${D}${libexecdir}/${PN}/stap-env
+   fi
+}
+
 BBCLASSEXTEND = "nativesdk"
-- 
2.7.4

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH v3 2/5] Revert "systemtap: Cross compilation fix"

[Victor Kamensky]

This reverts commit 787bed708676fc04aee2850825e803273152f657.

Signed-off-by: Victor Kamensky <kamen...@cisco.com>
---
 .../systemtap/systemtap/system_map_location.patch  | 23 --
 meta/recipes-kernel/systemtap/systemtap_git.inc|  1 -
 2 files changed, 24 deletions(-)
 delete mode 100644 
meta/recipes-kernel/systemtap/systemtap/system_map_location.patch

diff --git a/meta/recipes-kernel/systemtap/systemtap/system_map_location.patch 
b/meta/recipes-kernel/systemtap/systemtap/system_map_location.patch
deleted file mode 100644
index 013af5c..000
--- a/meta/recipes-kernel/systemtap/systemtap/system_map_location.patch
+++ /dev/null
@@ -1,23 +0,0 @@
-systemtap: Cross compilation fix
-
-This is a cross compilation fix. It allows systemtap to find
-the kernel map file in the right place, i.e. in the kernel build tree.
-Without this fix it takes a map file from the build host, if available.
-
-Upstream-Status: Pending
-
-Signed-off-by: Mikhail Durnev <mikhail_dur...@mentor.com>
-
-Index: git/session.cxx
-===
 git.orig/session.cxx
-+++ git/session.cxx
-@@ -1634,7 +1634,7 @@ systemtap_session::parse_kernel_function
-   clog << _F("Kernel symbol table %s unavailable, (%s)",
-  system_map_path.c_str(), strerror(errno)) << endl;
- 
--  system_map_path = "/boot/System.map-" + kernel_release;
-+  system_map_path = kernel_build_tree + "/System.map-" + kernel_release;
-   system_map.clear();
-   system_map.open(system_map_path.c_str(), ifstream::in);
-   if (! system_map.is_open())
diff --git a/meta/recipes-kernel/systemtap/systemtap_git.inc 
b/meta/recipes-kernel/systemtap/systemtap_git.inc
index 4d887ed..eef0c9c 100644
--- a/meta/recipes-kernel/systemtap/systemtap_git.inc
+++ b/meta/recipes-kernel/systemtap/systemtap_git.inc
@@ -4,7 +4,6 @@ SRCREV = "4051c70c9318c837981384cbb23f3e9eb1bd0892"
 PV = "3.2"
 
 SRC_URI = "git://sourceware.org/git/systemtap.git \
-   file://system_map_location.patch \
file://configure-allow-to-disable-libvirt.patch \
file://x32_abi_time.patch \
file://monitor-option.patch \
-- 
2.7.4

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH v3 3/5] systemtap: support --sysroot option in variety of situations in cross build

[Victor Kamensky]

For details on issues fixed please look at commit message of individual
patches.

Upstream-Status: Backport [system...@sourceware.org]
Signed-off-by: Victor Kamensky <kamen...@cisco.com>
---
 ...001-Added-a-couple-of-small-sysroot-fixes.patch |  42 +++
 ...g-sysroot-path-to-module-name-in-case-of-.patch |  61 ++
 ...sure-sysroot-paths-don-t-end-with-a-slash.patch | 128 +
 ...root-when-looking-for-the-System.map-file.patch |  29 +
 ...e_relocate-needs-target-file-path-not-hos.patch |  39 +++
 ...ookup-with-sysroot-case-do-not-remove-sys.patch |  42 +++
 ...-short-release-r-option-handling-follow-u.patch |  40 +++
 ...sroot-fix-short-release-r-option-handling.patch |  53 +
 ...dle-symbolic-links-with-absolute-name-rel.patch | 117 +++
 meta/recipes-kernel/systemtap/systemtap_git.inc|   9 ++
 10 files changed, 560 insertions(+)
 create mode 100644 
meta/recipes-kernel/systemtap/systemtap/0001-Added-a-couple-of-small-sysroot-fixes.patch
 create mode 100644 
meta/recipes-kernel/systemtap/systemtap/0001-Delay-adding-sysroot-path-to-module-name-in-case-of-.patch
 create mode 100644 
meta/recipes-kernel/systemtap/systemtap/0001-Make-sure-sysroot-paths-don-t-end-with-a-slash.patch
 create mode 100644 
meta/recipes-kernel/systemtap/systemtap/0001-Use-sysroot-when-looking-for-the-System.map-file.patch
 create mode 100644 
meta/recipes-kernel/systemtap/systemtap/0001-_stp_umodule_relocate-needs-target-file-path-not-hos.patch
 create mode 100644 
meta/recipes-kernel/systemtap/systemtap/0001-debuginfo-lookup-with-sysroot-case-do-not-remove-sys.patch
 create mode 100644 
meta/recipes-kernel/systemtap/systemtap/0001-sysroot-fix-short-release-r-option-handling-follow-u.patch
 create mode 100644 
meta/recipes-kernel/systemtap/systemtap/0001-sysroot-fix-short-release-r-option-handling.patch
 create mode 100644 
meta/recipes-kernel/systemtap/systemtap/0001-sysroot-handle-symbolic-links-with-absolute-name-rel.patch

diff --git 
a/meta/recipes-kernel/systemtap/systemtap/0001-Added-a-couple-of-small-sysroot-fixes.patch
 
b/meta/recipes-kernel/systemtap/systemtap/0001-Added-a-couple-of-small-sysroot-fixes.patch
new file mode 100644
index 000..c0ceb5a
--- /dev/null
+++ 
b/meta/recipes-kernel/systemtap/systemtap/0001-Added-a-couple-of-small-sysroot-fixes.patch
@@ -0,0 +1,42 @@
+From a714658727206d2a98a7194b7e6d29dbd3e27b8d Mon Sep 17 00:00:00 2001
+From: David Smith <dsm...@redhat.com>
+Date: Mon, 19 Mar 2018 16:50:05 -0500
+Subject: [PATCH] Added a couple of small sysroot fixes.
+
+* tapsets.cxx (dwarf_builder::build): Fix commit 4ffecddf5.
+  (path_remove_sysroot): Fix extra '/' present at start of paths.
+
+Upstream-Status: Backport
+Signed-off-by: Victor Kamensky <kamen...@cisco.com>
+---
+ tapsets.cxx | 10 +++---
+ 1 file changed, 7 insertions(+), 3 deletions(-)
+
+Index: git/tapsets.cxx
+===
+--- git.orig/tapsets.cxx
 git/tapsets.cxx
+@@ -1395,7 +1395,8 @@ string path_remove_sysroot(const systemt
+   string retval = path;
+   if (!sess.sysroot.empty() &&
+   (pos = retval.find(sess.sysroot)) != string::npos)
+-retval.replace(pos, sess.sysroot.length(), "/");
++retval.replace(pos, sess.sysroot.length(),
++ (sess.sysroot.back() == '/' ? "/": ""));
+   return retval;
+ }
+ 
+@@ -8412,8 +8413,11 @@ dwarf_builder::build(systemtap_session &
+ 
+   // PR13338: unquote glob results
+   module_name = unescape_glob_chars (module_name);
+-  user_path = find_executable (module_name, "", sess.sysenv); // 
canonicalize it
+-  if (!is_fully_resolved(user_path, sess.sysroot, sess.sysenv))
++  user_path = find_executable (module_name, sess.sysroot, sess.sysenv); 
// canonicalize it
++  // Note we don't need to pass the sysroot to
++  // is_fully_resolved(), since we just passed it to
++  // find_executable().
++  if (!is_fully_resolved(user_path, "", sess.sysenv))
+ throw SEMANTIC_ERROR(_F("cannot find executable '%s'",
+ user_path.to_string().c_str()));
+ 
diff --git 
a/meta/recipes-kernel/systemtap/systemtap/0001-Delay-adding-sysroot-path-to-module-name-in-case-of-.patch
 
b/meta/recipes-kernel/systemtap/systemtap/0001-Delay-adding-sysroot-path-to-module-name-in-case-of-.patch
new file mode 100644
index 000..89951a2
--- /dev/null
+++ 
b/meta/recipes-kernel/systemtap/systemtap/0001-Delay-adding-sysroot-path-to-module-name-in-case-of-.patch
@@ -0,0 +1,61 @@
+From 4ffecddf5433d65a6f01241990c9d516586b1c79 Mon Sep 17 00:00:00 2001
+From: Victor Kamensky <kamen...@cisco.com>
+Date: Mon, 19 Mar 2018 08:53:51 -0500
+Subject: [PATCH] Delay adding sysroot path to module name in case of non
+ absolute executable
+
+Current stap code adds sysroot prematurely for probes that specify non
+a

[OE-core] [PATCH v3 1/5] image: add image-combined-dbg helper class

[Victor Kamensky]

There is IMAGE_GEN_DEBUGFS="1" variable that enables build of
additional rootfs-dbg and additional archive that contains
complimentary symbols files for a given image. But the issue
with this resulting directory and tarball that before use it
has to be combined with original image content. It is required
since all cross debugging tools like gdb, perf, and systemtap
need file system that contains both target executables/libraries
and their symbols. Those tools need to find executable/library
first and through it debuglink note find corresponding symbols
file.

image-combined-dbg when added to USER_CLASSES just copies
final resulting rootfs back into rootfs-dbg creating combined
target and debug symbols rootfs that can be used for debugging
directly.

Signed-off-by: Victor Kamensky <kamen...@cisco.com>
---
 meta/classes/image-combined-dbg.bbclass | 9 +
 1 file changed, 9 insertions(+)
 create mode 100644 meta/classes/image-combined-dbg.bbclass

diff --git a/meta/classes/image-combined-dbg.bbclass 
b/meta/classes/image-combined-dbg.bbclass
new file mode 100644
index 000..f4772f7
--- /dev/null
+++ b/meta/classes/image-combined-dbg.bbclass
@@ -0,0 +1,9 @@
+IMAGE_PREPROCESS_COMMAND_append = " combine_dbg_image; "
+
+combine_dbg_image () {
+if [ "${IMAGE_GEN_DEBUGFS}" = "1" -a -e ${IMAGE_ROOTFS}-dbg ]; then
+# copy target files into -dbg rootfs, so it can be used for
+# debug purposes directly
+tar -C ${IMAGE_ROOTFS} -cf - . | tar -C ${IMAGE_ROOTFS}-dbg 
-xf -
+fi
+}
-- 
2.7.4

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH v3 0/5] systemtap adding sysroot, cross compiling of user land related scripts

[Victor Kamensky]

Hi,

This is third version of patch series of SystemTap supporting
sysroot and SystemTap script cross compilation, integrated with
OE workflow.

Version 3 Changes:

   * Simplified and cleaned image-combined-dbg class, as per Richard's
 neat suggestion

   * Rebased to latest tree to resolve merge conflict with systemtap
 and gcc-8 recent change

Version 2 Patches, since first, RFC, version [1] the following changes
were done:

   image: add image-combined-dbg helper class

  Needed to create rootfs-dbg that contains both target binaries
  and debug symbols. Instead of IMAGE_GEN_COMBINED_DEBUGFS build
  option as in RFC patches [2], small additional class is
  introduced that allows adding target symbols back into
  rootfs-dbg. The approach was suggested by Khem, as response on
  first RFC patches.

   Revert "systemtap: Cross compilation fix"

  Remove previous OE specific patch. It superseded
  by --sysroot support patch series

   systemtap: support --sysroot option in variety of situations in cross build

  Backport of SystemTap patches required to support --sysroot.
  Since my first post I worked with SystemTap maintainers and
  now all needed fixes for sysroot support got accepted.

   systemtap: create translator packageconfig

  Creates new PACKAGECONFIG, translator, which is on by default,
  but if dropped resulting systemtap package would contain only
  minimal run-time utilities required to activate/run host cross
  compiled SystemTap modules. It is implemented in such way
  instead of separate systemtap-utils recipe based on team's
  feedback.

   crosstap: replace script with new python based implementation

  Full backward compatible replacement of original crosstap
  shell script, written in python with extended functionality
  to support SystemTap scripts for user-land using --sysroot
  option. Compared to RFC version new script invocation wise is
  fully backward compatible with orginal crosstap shell script.

Full setup with patch series and examples that were used for
testing is available at [3].

[1] 
http://lists.openembedded.org/pipermail/openembedded-core/2018-March/148387.html

[2] 
http://lists.openembedded.org/pipermail/openembedded-core/2018-March/148392.html

[3] https://github.com/victorkamensky/systemtap-oe-sysroot-manifest

Victor Kamensky (5):
  image: add image-combined-dbg helper class
  Revert "systemtap: Cross compilation fix"
  systemtap: support --sysroot option in variety of situations in cross
build
  systemtap: create translator packageconfig
  crosstap: replace script with new python based implementation

 meta/classes/image-combined-dbg.bbclass|   9 +
 ...001-Added-a-couple-of-small-sysroot-fixes.patch |  42 ++
 ...g-sysroot-path-to-module-name-in-case-of-.patch |  61 +++
 ...sure-sysroot-paths-don-t-end-with-a-slash.patch | 128 +
 ...root-when-looking-for-the-System.map-file.patch |  29 +
 ...e_relocate-needs-target-file-path-not-hos.patch |  39 ++
 ...ookup-with-sysroot-case-do-not-remove-sys.patch |  42 ++
 ...-short-release-r-option-handling-follow-u.patch |  40 ++
 ...sroot-fix-short-release-r-option-handling.patch |  53 ++
 ...dle-symbolic-links-with-absolute-name-rel.patch | 117 
 .../systemtap/systemtap/system_map_location.patch  |  23 -
 meta/recipes-kernel/systemtap/systemtap_git.bb |  15 +-
 meta/recipes-kernel/systemtap/systemtap_git.inc|  10 +-
 scripts/crosstap   | 586 -
 14 files changed, 1030 insertions(+), 164 deletions(-)
 create mode 100644 meta/classes/image-combined-dbg.bbclass
 create mode 100644 
meta/recipes-kernel/systemtap/systemtap/0001-Added-a-couple-of-small-sysroot-fixes.patch
 create mode 100644 
meta/recipes-kernel/systemtap/systemtap/0001-Delay-adding-sysroot-path-to-module-name-in-case-of-.patch
 create mode 100644 
meta/recipes-kernel/systemtap/systemtap/0001-Make-sure-sysroot-paths-don-t-end-with-a-slash.patch
 create mode 100644 
meta/recipes-kernel/systemtap/systemtap/0001-Use-sysroot-when-looking-for-the-System.map-file.patch
 create mode 100644 
meta/recipes-kernel/systemtap/systemtap/0001-_stp_umodule_relocate-needs-target-file-path-not-hos.patch
 create mode 100644 
meta/recipes-kernel/systemtap/systemtap/0001-debuginfo-lookup-with-sysroot-case-do-not-remove-sys.patch
 create mode 100644 
meta/recipes-kernel/systemtap/systemtap/0001-sysroot-fix-short-release-r-option-handling-follow-u.patch
 create mode 100644 
meta/recipes-kernel/systemtap/systemtap/0001-sysroot-fix-short-release-r-option-handling.patch
 create mode 100644 
meta/recipes-kernel/systemtap/systemtap/0001-sysroot-handle-symbolic-links-with-absolute-name-rel.patch
 delete mode 100644 
meta/recipes-kernel/systemtap/systemtap/system_map_location.patch

-- 
2.7.4

-- 
___
Openembedded-core mailing list
Openembedded-core@li

[OE-core] [PATCH v2 1/5] image: add image-combined-dbg helper class

[Victor Kamensky]

There is IMAGE_GEN_DEBUGFS="1" variable that enables build of
additional rootfs-dbg and additional archive that contains
complimentary symbols files for a given image. But the issue
with this resulting directory and tarball that before use it
has to be combined with original image content. It is required
since all cross debugging tools like gdb, perf, and systemtap
need file system that contains both target executables/libraries
and their symbols. Those tools need to find executable/library
first and through it debuglink note find corresponding symbols
file.

image-combined-dbg when added to USER_CLASSES just copies
final resulting rootfs back into rootfs-dbg creating combined
target and debug symbols rootfs that can be used for debugging
directly.

Signed-off-by: Victor Kamensky <kamen...@cisco.com>
---
 meta/classes/image-combined-dbg.bbclass | 9 +
 meta/classes/image.bbclass  | 1 +
 2 files changed, 10 insertions(+)
 create mode 100644 meta/classes/image-combined-dbg.bbclass

diff --git a/meta/classes/image-combined-dbg.bbclass 
b/meta/classes/image-combined-dbg.bbclass
new file mode 100644
index 000..a709adf
--- /dev/null
+++ b/meta/classes/image-combined-dbg.bbclass
@@ -0,0 +1,9 @@
+IMAGE_PREPROCESS_COMMAND_append = " combine_dbg_image; "
+
+combine_dbg_image () {
+if [ "$IMAGE_GEN_DEBUGFS" = "1" -a -e ${IMAGE_ROOTFS}-dbg ]; then
+# copy target files into -dbg rootfs, so it can be used for
+# debug purposes directly
+tar -C ${IMAGE_ROOTFS} -cf - . | tar -C ${IMAGE_ROOTFS}-dbg 
-xf -
+fi
+}
diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index 1636eed..5ee9c55 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -268,6 +268,7 @@ fakeroot python do_image () {
 from oe.utils import execute_pre_post_process
 
 d.setVarFlag('REPRODUCIBLE_TIMESTAMP_ROOTFS', 'export', '1')
+d.setVarFlag('IMAGE_GEN_DEBUGFS', 'export', '1')
 pre_process_cmds = d.getVar("IMAGE_PREPROCESS_COMMAND")
 
 execute_pre_post_process(d, pre_process_cmds)
-- 
2.7.4

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH v2 2/5] Revert "systemtap: Cross compilation fix"

[Victor Kamensky]

This reverts commit 787bed708676fc04aee2850825e803273152f657.

Signed-off-by: Victor Kamensky <kamen...@cisco.com>
---
 .../systemtap/systemtap/system_map_location.patch  | 23 --
 meta/recipes-kernel/systemtap/systemtap_git.inc|  1 -
 2 files changed, 24 deletions(-)
 delete mode 100644 
meta/recipes-kernel/systemtap/systemtap/system_map_location.patch

diff --git a/meta/recipes-kernel/systemtap/systemtap/system_map_location.patch 
b/meta/recipes-kernel/systemtap/systemtap/system_map_location.patch
deleted file mode 100644
index 013af5c..000
--- a/meta/recipes-kernel/systemtap/systemtap/system_map_location.patch
+++ /dev/null
@@ -1,23 +0,0 @@
-systemtap: Cross compilation fix
-
-This is a cross compilation fix. It allows systemtap to find
-the kernel map file in the right place, i.e. in the kernel build tree.
-Without this fix it takes a map file from the build host, if available.
-
-Upstream-Status: Pending
-
-Signed-off-by: Mikhail Durnev <mikhail_dur...@mentor.com>
-
-Index: git/session.cxx
-===
 git.orig/session.cxx
-+++ git/session.cxx
-@@ -1634,7 +1634,7 @@ systemtap_session::parse_kernel_function
-   clog << _F("Kernel symbol table %s unavailable, (%s)",
-  system_map_path.c_str(), strerror(errno)) << endl;
- 
--  system_map_path = "/boot/System.map-" + kernel_release;
-+  system_map_path = kernel_build_tree + "/System.map-" + kernel_release;
-   system_map.clear();
-   system_map.open(system_map_path.c_str(), ifstream::in);
-   if (! system_map.is_open())
diff --git a/meta/recipes-kernel/systemtap/systemtap_git.inc 
b/meta/recipes-kernel/systemtap/systemtap_git.inc
index f51bd28..af5b004 100644
--- a/meta/recipes-kernel/systemtap/systemtap_git.inc
+++ b/meta/recipes-kernel/systemtap/systemtap_git.inc
@@ -4,7 +4,6 @@ SRCREV = "4051c70c9318c837981384cbb23f3e9eb1bd0892"
 PV = "3.2"
 
 SRC_URI = "git://sourceware.org/git/systemtap.git \
-   file://system_map_location.patch \
file://configure-allow-to-disable-libvirt.patch \
file://x32_abi_time.patch \
file://monitor-option.patch \
-- 
2.7.4

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH v2 4/5] systemtap: create translator packageconfig

[Victor Kamensky]

For cases when systemap module compilation happens on host in
cross-compilation mode, and it is desirable to minimize systemtap
presense on target we need to have just smallest possible set of
utilties that are required to run compiled modules.

Introduce new "translator" PACKAGECONFIG, if it is not set
it would mean that just minimal set of run-time utilities will
be included in the package.

For run-time only systemtap build variant use
PACKAGECONFIG_pn-systemtap = "" or
PACKAGECONFIG_pn-systemtap = "monitor"

Suggested-by: Taras Kondratiuk <takon...@cisco.com>
Signed-off-by: Victor Kamensky <kamen...@cisco.com>
---
 meta/recipes-kernel/systemtap/systemtap_git.bb | 15 +++
 1 file changed, 11 insertions(+), 4 deletions(-)

diff --git a/meta/recipes-kernel/systemtap/systemtap_git.bb 
b/meta/recipes-kernel/systemtap/systemtap_git.bb
index 475b207..b280f58 100644
--- a/meta/recipes-kernel/systemtap/systemtap_git.bb
+++ b/meta/recipes-kernel/systemtap/systemtap_git.bb
@@ -3,9 +3,7 @@ HOMEPAGE = "https://sourceware.org/systemtap/;
 
 require systemtap_git.inc
 
-DEPENDS = "boost elfutils"
-
-RDEPENDS_${PN} += "python3-core bash perl"
+DEPENDS = "elfutils"
 
 EXTRA_OECONF += "--with-libelf=${STAGING_DIR_TARGET} --without-rpm \
 --without-nss --without-avahi --without-dyninst \
@@ -18,7 +16,8 @@ STAP_DOCS ?= "--disable-docs --disable-publican 
--disable-refdocs"
 
 EXTRA_OECONF += "${STAP_DOCS} "
 
-PACKAGECONFIG ??= "sqlite monitor python3-probes"
+PACKAGECONFIG ??= "translator sqlite monitor python3-probes"
+PACKAGECONFIG[translator] = 
"--enable-translator,--disable-translator,boost,python3-core bash perl"
 PACKAGECONFIG[libvirt] = "--enable-libvirt,--disable-libvirt,libvirt"
 PACKAGECONFIG[sqlite] = "--enable-sqlite,--disable-sqlite,sqlite3"
 PACKAGECONFIG[monitor] = "--enable-monitor,--disable-monitor,ncurses json-c"
@@ -26,4 +25,12 @@ PACKAGECONFIG[python3-probes] = 
"--with-python3-probes,--without-python3-probes,
 
 inherit autotools gettext pkgconfig distutils3-base
 
+do_install_append () {
+   if [ ! -f ${D}${bindir}/stap ]; then
+  # translator disabled case, need to leave only minimal runtime
+  rm -rf ${D}${datadir}/${PN}
+  rm ${D}${libexecdir}/${PN}/stap-env
+   fi
+}
+
 BBCLASSEXTEND = "nativesdk"
-- 
2.7.4

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH v2 3/5] systemtap: support --sysroot option in variety of situations in cross build

[Victor Kamensky]

For details on issues fixed please look at commit message of individual
patches.

Upstream-Status: Backport [system...@sourceware.org]
Signed-off-by: Victor Kamensky <kamen...@cisco.com>
---
 ...001-Added-a-couple-of-small-sysroot-fixes.patch |  42 +++
 ...g-sysroot-path-to-module-name-in-case-of-.patch |  61 ++
 ...sure-sysroot-paths-don-t-end-with-a-slash.patch | 128 +
 ...root-when-looking-for-the-System.map-file.patch |  29 +
 ...e_relocate-needs-target-file-path-not-hos.patch |  39 +++
 ...ookup-with-sysroot-case-do-not-remove-sys.patch |  42 +++
 ...-short-release-r-option-handling-follow-u.patch |  40 +++
 ...sroot-fix-short-release-r-option-handling.patch |  53 +
 ...dle-symbolic-links-with-absolute-name-rel.patch | 117 +++
 meta/recipes-kernel/systemtap/systemtap_git.inc|   9 ++
 10 files changed, 560 insertions(+)
 create mode 100644 
meta/recipes-kernel/systemtap/systemtap/0001-Added-a-couple-of-small-sysroot-fixes.patch
 create mode 100644 
meta/recipes-kernel/systemtap/systemtap/0001-Delay-adding-sysroot-path-to-module-name-in-case-of-.patch
 create mode 100644 
meta/recipes-kernel/systemtap/systemtap/0001-Make-sure-sysroot-paths-don-t-end-with-a-slash.patch
 create mode 100644 
meta/recipes-kernel/systemtap/systemtap/0001-Use-sysroot-when-looking-for-the-System.map-file.patch
 create mode 100644 
meta/recipes-kernel/systemtap/systemtap/0001-_stp_umodule_relocate-needs-target-file-path-not-hos.patch
 create mode 100644 
meta/recipes-kernel/systemtap/systemtap/0001-debuginfo-lookup-with-sysroot-case-do-not-remove-sys.patch
 create mode 100644 
meta/recipes-kernel/systemtap/systemtap/0001-sysroot-fix-short-release-r-option-handling-follow-u.patch
 create mode 100644 
meta/recipes-kernel/systemtap/systemtap/0001-sysroot-fix-short-release-r-option-handling.patch
 create mode 100644 
meta/recipes-kernel/systemtap/systemtap/0001-sysroot-handle-symbolic-links-with-absolute-name-rel.patch

diff --git 
a/meta/recipes-kernel/systemtap/systemtap/0001-Added-a-couple-of-small-sysroot-fixes.patch
 
b/meta/recipes-kernel/systemtap/systemtap/0001-Added-a-couple-of-small-sysroot-fixes.patch
new file mode 100644
index 000..c0ceb5a
--- /dev/null
+++ 
b/meta/recipes-kernel/systemtap/systemtap/0001-Added-a-couple-of-small-sysroot-fixes.patch
@@ -0,0 +1,42 @@
+From a714658727206d2a98a7194b7e6d29dbd3e27b8d Mon Sep 17 00:00:00 2001
+From: David Smith <dsm...@redhat.com>
+Date: Mon, 19 Mar 2018 16:50:05 -0500
+Subject: [PATCH] Added a couple of small sysroot fixes.
+
+* tapsets.cxx (dwarf_builder::build): Fix commit 4ffecddf5.
+  (path_remove_sysroot): Fix extra '/' present at start of paths.
+
+Upstream-Status: Backport
+Signed-off-by: Victor Kamensky <kamen...@cisco.com>
+---
+ tapsets.cxx | 10 +++---
+ 1 file changed, 7 insertions(+), 3 deletions(-)
+
+Index: git/tapsets.cxx
+===
+--- git.orig/tapsets.cxx
 git/tapsets.cxx
+@@ -1395,7 +1395,8 @@ string path_remove_sysroot(const systemt
+   string retval = path;
+   if (!sess.sysroot.empty() &&
+   (pos = retval.find(sess.sysroot)) != string::npos)
+-retval.replace(pos, sess.sysroot.length(), "/");
++retval.replace(pos, sess.sysroot.length(),
++ (sess.sysroot.back() == '/' ? "/": ""));
+   return retval;
+ }
+ 
+@@ -8412,8 +8413,11 @@ dwarf_builder::build(systemtap_session &
+ 
+   // PR13338: unquote glob results
+   module_name = unescape_glob_chars (module_name);
+-  user_path = find_executable (module_name, "", sess.sysenv); // 
canonicalize it
+-  if (!is_fully_resolved(user_path, sess.sysroot, sess.sysenv))
++  user_path = find_executable (module_name, sess.sysroot, sess.sysenv); 
// canonicalize it
++  // Note we don't need to pass the sysroot to
++  // is_fully_resolved(), since we just passed it to
++  // find_executable().
++  if (!is_fully_resolved(user_path, "", sess.sysenv))
+ throw SEMANTIC_ERROR(_F("cannot find executable '%s'",
+ user_path.to_string().c_str()));
+ 
diff --git 
a/meta/recipes-kernel/systemtap/systemtap/0001-Delay-adding-sysroot-path-to-module-name-in-case-of-.patch
 
b/meta/recipes-kernel/systemtap/systemtap/0001-Delay-adding-sysroot-path-to-module-name-in-case-of-.patch
new file mode 100644
index 000..89951a2
--- /dev/null
+++ 
b/meta/recipes-kernel/systemtap/systemtap/0001-Delay-adding-sysroot-path-to-module-name-in-case-of-.patch
@@ -0,0 +1,61 @@
+From 4ffecddf5433d65a6f01241990c9d516586b1c79 Mon Sep 17 00:00:00 2001
+From: Victor Kamensky <kamen...@cisco.com>
+Date: Mon, 19 Mar 2018 08:53:51 -0500
+Subject: [PATCH] Delay adding sysroot path to module name in case of non
+ absolute executable
+
+Current stap code adds sysroot prematurely for probes that specify non
+a

[OE-core] [PATCH v2 5/5] crosstap: replace script with new python based implementation

[Victor Kamensky]

New crosstap python implementation is total replacement for
crosstap shell script, that has superseding capabilities.
New script support cross compiling of SystemTap scripts
for user-land, by using supplied image rootfs. Whereas old
script could only deal with scripts against kernel. New script
has more complex logic and additional capabilities.

As invocation interface new script support old "legacy"
mode and provides alternative new regular options interface
to access additional functionality.

Signed-off-by: Victor Kamensky <kamen...@cisco.com>
---
 scripts/crosstap | 586 ++-
 1 file changed, 450 insertions(+), 136 deletions(-)

diff --git a/scripts/crosstap b/scripts/crosstap
index 39739bb..e33fa4a 100755
--- a/scripts/crosstap
+++ b/scripts/crosstap
@@ -1,15 +1,22 @@
-#!/bin/bash
+#!/usr/bin/env python3
 #
-# Run a systemtap script on remote target
+# Build a systemtap script for a given image, kernel
 #
-# Examples (run on build host, target is 192.168.1.xxx):
-#   $ source oe-init-build-env"
-#   $ cd ~/my/systemtap/scripts"
+# Effectively script extracts needed information from set of
+# 'bitbake -e' commands and contructs proper invocation of stap on
+# host to build systemtap script for a given target.
 #
-#   $ crosstap r...@192.168.1.xxx myscript.stp"
-#   $ crosstap r...@192.168.1.xxx myscript-with-args.stp 99 ninetynine"
+# By default script will compile scriptname.ko that could be copied
+# to taget and activated with 'staprun scriptname.ko' command. Or if
+# --remote user@hostname option is specified script will build, load
+# execute script on target.
 #
-# Copyright (c) 2012, Intel Corporation.
+# This script is very similar and inspired by crosstap shell script.
+# The major difference that this script supports user-land related
+# systemtap script, whereas crosstap could deal only with scripts
+# related to kernel.
+#
+# Copyright (c) 2018, Cisco Systems.
 # All rights reserved.
 #
 # This program is free software; you can redistribute it and/or modify
@@ -25,131 +32,438 @@
 # along with this program; if not, write to the Free Software
 # Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
 
-function usage() {
-echo "Usage: $0 <user@hostname>  [additional 
systemtap-script args]"
-}
-
-function setup_usage() {
-echo ""
-echo "'crosstap' requires a local sdk build of the target system"
-echo "(or a build that includes 'tools-profile') in order to build"
-echo "kernel modules that can probe the target system."
-echo ""
-echo "Practically speaking, that means you need to do the following:"
-echo "  - If you're running a pre-built image, download the release"
-echo "and/or BSP tarballs used to build the image."
-echo "  - If you're working from git sources, just clone the metadata"
-echo "and BSP layers needed to build the image you'll be booting."
-echo "  - Make sure you're properly set up to build a new image (see"
-echo "the BSP README and/or the widely available basic documentation"
-echo "that discusses how to build images)."
-echo "  - Build an -sdk version of the image e.g.:"
-echo "  $ bitbake core-image-sato-sdk"
-echo "  OR"
-echo "  - Build a non-sdk image but include the profiling tools:"
-echo "  [ edit local.conf and add 'tools-profile' to the end of"
-echo "the EXTRA_IMAGE_FEATURES variable ]"
-echo "  $ bitbake core-image-sato"
-echo ""
-echo "  [ NOTE that 'crosstap' needs to be able to ssh into the target"
-echo "system, which isn't enabled by default in -minimal images. ]"
-echo ""
-echo "Once you've build the image on the host system, you're ready to"
-echo "boot it (or the equivalent pre-built image) and use 'crosstap'"
-echo "to probe it (you need to source the environment as usual first):"
-echo ""
-echo "$ source oe-init-build-env"
-echo "$ cd ~/my/systemtap/scripts"
-echo "$ crosstap r...@192.168.1.xxx myscript.stp"
-echo ""
-}
-
-function systemtap_target_arch() {
-SYSTEMTAP_TARGET_ARCH=$1
-case $SYSTEMTAP_TARGET_ARCH in
-i?86)
-SYSTEMTAP_TARGET_ARCH="i386"
-;;
-x86?64*)
-SYSTEMTAP_TARGET_ARCH="x86_64"
-;;
-arm*)
-SYSTEMTAP_TARGET_ARCH="arm"
-;;
-powerpc*)
-SYSTEMTAP_TARGET_ARCH="powerpc"
-;;
-*)
-;;
-esac
-}
-
-if [ $# -lt 2 ]; then
-   u

[OE-core] [PATCH v2 0/5] systemtap adding sysroot, cross compiling of user land related scripts

[Victor Kamensky]

Hi,

This is second version of patch series of SystemTap supporting
sysroot and SystemTap script cross compilation, integrated with
OE workflow.

Since first, RFC, version [1] the following changes were done:

   image: add image-combined-dbg helper class

  Needed to create rootfs-dbg that contains both target binaries
  and debug symbols. Instead of IMAGE_GEN_COMBINED_DEBUGFS build
  option as in RFC patches [2], small additional class is
  introduced that allows adding target symbols back into
  rootfs-dbg. The approach was suggested by Khem, as response on
  first RFC patches.

   Revert "systemtap: Cross compilation fix"

  Remove previous OE specific patch. It superseded
  by --sysroot support patch series

   systemtap: support --sysroot option in variety of situations in cross build

  Backport of SystemTap patches required to support --sysroot.
  Since my first post I worked with SystemTap maintainers and
  now all needed fixes for sysroot support got accepted.

   systemtap: create translator packageconfig

  Creates new PACKAGECONFIG, translator, which is on by default,
  but if dropped resulting systemtap package would contain only
  minimal run-time utilities required to activate/run host cross
  compiled SystemTap modules. It is implemented in such way
  instead of separate systemtap-utils recipe based on team's
  feedback.

   crosstap: replace script with new python based implementation

  Full backward compatible replacement of original crosstap
  shell script, written in python with extended functionality
  to support SystemTap scripts for user-land using --sysroot
  option. Compared to RFC version new script invocation wise is
  fully backward compatible with orginal crosstap shell script.

Full setup with patch series and examples that were used for
testing is available at [3].

[1] 
http://lists.openembedded.org/pipermail/openembedded-core/2018-March/148387.html

[2] 
http://lists.openembedded.org/pipermail/openembedded-core/2018-March/148392.html

[3] https://github.com/victorkamensky/systemtap-oe-sysroot-manifest

Victor Kamensky (5):
  image: add image-combined-dbg helper class
  Revert "systemtap: Cross compilation fix"
  systemtap: support --sysroot option in variety of situations in cross
build
  systemtap: create translator packageconfig
  crosstap: replace script with new python based implementation

 meta/classes/image-combined-dbg.bbclass|   9 +
 meta/classes/image.bbclass |   1 +
 ...001-Added-a-couple-of-small-sysroot-fixes.patch |  42 ++
 ...g-sysroot-path-to-module-name-in-case-of-.patch |  61 +++
 ...sure-sysroot-paths-don-t-end-with-a-slash.patch | 128 +
 ...root-when-looking-for-the-System.map-file.patch |  29 +
 ...e_relocate-needs-target-file-path-not-hos.patch |  39 ++
 ...ookup-with-sysroot-case-do-not-remove-sys.patch |  42 ++
 ...-short-release-r-option-handling-follow-u.patch |  40 ++
 ...sroot-fix-short-release-r-option-handling.patch |  53 ++
 ...dle-symbolic-links-with-absolute-name-rel.patch | 117 
 .../systemtap/systemtap/system_map_location.patch  |  23 -
 meta/recipes-kernel/systemtap/systemtap_git.bb |  15 +-
 meta/recipes-kernel/systemtap/systemtap_git.inc|  10 +-
 scripts/crosstap   | 586 -
 15 files changed, 1031 insertions(+), 164 deletions(-)
 create mode 100644 meta/classes/image-combined-dbg.bbclass
 create mode 100644 
meta/recipes-kernel/systemtap/systemtap/0001-Added-a-couple-of-small-sysroot-fixes.patch
 create mode 100644 
meta/recipes-kernel/systemtap/systemtap/0001-Delay-adding-sysroot-path-to-module-name-in-case-of-.patch
 create mode 100644 
meta/recipes-kernel/systemtap/systemtap/0001-Make-sure-sysroot-paths-don-t-end-with-a-slash.patch
 create mode 100644 
meta/recipes-kernel/systemtap/systemtap/0001-Use-sysroot-when-looking-for-the-System.map-file.patch
 create mode 100644 
meta/recipes-kernel/systemtap/systemtap/0001-_stp_umodule_relocate-needs-target-file-path-not-hos.patch
 create mode 100644 
meta/recipes-kernel/systemtap/systemtap/0001-debuginfo-lookup-with-sysroot-case-do-not-remove-sys.patch
 create mode 100644 
meta/recipes-kernel/systemtap/systemtap/0001-sysroot-fix-short-release-r-option-handling-follow-u.patch
 create mode 100644 
meta/recipes-kernel/systemtap/systemtap/0001-sysroot-fix-short-release-r-option-handling.patch
 create mode 100644 
meta/recipes-kernel/systemtap/systemtap/0001-sysroot-handle-symbolic-links-with-absolute-name-rel.patch
 delete mode 100644 
meta/recipes-kernel/systemtap/systemtap/system_map_location.patch

-- 
2.7.4

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH v2] oeqa/runtime/stap.py: add runtime test for systemtap

[Victor Kamensky]

Add runtime test for stap to test basic SystemTap
operations: can compile very basic module and run on
target device.

Note we disable (-DSTP_NO_VERREL_CHECK) SystemTap
additional kernel release check since during OE testing
mismatching kernel-devsrc and kernels are used.

Signed-off-by: Victor Kamensky <kamen...@cisco.com>
---

 v2 updates:
* prepare in /usr/src/kernel make as in
  kernelmodule.py

* use of uname -r for kernel release retrieval

* adds -DSTP_NO_VERREL_CHECK to allow mismatch
  between running kernel and kernel-devsrc

 meta/lib/oeqa/runtime/cases/stap.py   | 33 +
 meta/lib/oeqa/runtime/files/hello.stp |  1 +
 2 files changed, 34 insertions(+)
 create mode 100644 meta/lib/oeqa/runtime/cases/stap.py
 create mode 100644 meta/lib/oeqa/runtime/files/hello.stp

diff --git a/meta/lib/oeqa/runtime/cases/stap.py 
b/meta/lib/oeqa/runtime/cases/stap.py
new file mode 100644
index 000..fc728bf
--- /dev/null
+++ b/meta/lib/oeqa/runtime/cases/stap.py
@@ -0,0 +1,33 @@
+import os
+
+from oeqa.runtime.case import OERuntimeTestCase
+from oeqa.core.decorator.depends import OETestDepends
+from oeqa.core.decorator.oeid import OETestID
+from oeqa.core.decorator.data import skipIfNotFeature
+
+class StapTest(OERuntimeTestCase):
+
+@classmethod
+def setUpClass(cls):
+src = os.path.join(cls.tc.runtime_files_dir, 'hello.stp')
+dst = '/tmp/hello.stp'
+cls.tc.target.copyTo(src, dst)
+
+@classmethod
+def tearDownClass(cls):
+files = '/tmp/hello.stp'
+cls.tc.target.run('rm %s' % files)
+
+@OETestID(1652)
+@skipIfNotFeature('tools-profile',
+  'Test requires tools-profile to be in IMAGE_FEATURES')
+@OETestDepends(['kernelmodule.KernelModuleTest.test_kernel_module'])
+def test_stap(self):
+cmds = [
+'cd /usr/src/kernel && make scripts prepare',
+'cd /lib/modules/`uname -r` && (if [ ! -L build ]; then ln -s 
/usr/src/kernel build; fi)',
+'stap --disable-cache -DSTP_NO_VERREL_CHECK /tmp/hello.stp'
+]
+for cmd in cmds:
+status, output = self.target.run(cmd, 900)
+self.assertEqual(status, 0, msg='\n'.join([cmd, output]))
diff --git a/meta/lib/oeqa/runtime/files/hello.stp 
b/meta/lib/oeqa/runtime/files/hello.stp
new file mode 100644
index 000..3677147
--- /dev/null
+++ b/meta/lib/oeqa/runtime/files/hello.stp
@@ -0,0 +1 @@
+probe oneshot { println("hello world") }
-- 
2.7.4

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH] oeqa/runtime/stap.py: add runtime test for systemtap

[Victor Kamensky]

On Mon, 26 Mar 2018, Burton, Ross wrote:


On the autobuilder:

| NOTE: FAIL [15.139s]: test_stap (stap.StapTest)
| NOTE: --
| NOTE: Traceback (most recent call last):
|   File 
"/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-rpm-non-rpm/build/meta/lib/oeqa/core/decorator/__init__.py",
line 32, in wrapped_f
| return func(*args, **kwargs)
|   File 
"/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-rpm-non-rpm/build/meta/lib/oeqa/core/decorator/__init__.py",
line 32, in wrapped_f
| return func(*args, **kwargs)
|   File 
"/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-rpm-non-rpm/build/meta/lib/oeqa/core/decorator/__init__.py",
line 32, in wrapped_f
| return func(*args, **kwargs)
|   File 
"/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-rpm-non-rpm/build/meta/lib/oeqa/runtime/cases/stap.py",
line 33, in test_stap
| self.assertEqual(status, 0, msg='\n'.join([cmd, output]))
| AssertionError: 1 != 0 : stap --disable-cache /tmp/hello.stp
| ERROR: module version mismatch (#1 SMP PREEMPT Mon Mar 26 14:41:21
UTC 2018 vs #1 SMP PREEMPT Mon Mar 26 14:03:27 GMT 2018), release
4.14.24-yocto-standard
| WARNING: /usr/bin/staprun exited with status: 1
| Pass 5: run failed.  [man error::pass5]

Is this systemtap-uprobes not being rebuilt when it should?


systemtap-uprobes should not be used nowdays.

Does anyone have pointer to what autobuilder is executing?

Above message indicates that version of kernel that is running
'uname -v' does not match version string recorded in
/usr/src/kernel/include/generated/compile.h (which is part of
kernel-devsrc package). Usually such version check is performed
by kernel CONFIG_MODVERSIONS, which is not enabled in OE build.
SystemTap specifically adds its own kernel version check (the
one firing above) and it could be disabled through command
line parameter.

It is easy to reproduce above error by building core-image-lsb-sdk,
then running 'bitbake -c cleansstate linux-yocto' and rebuilding
core-image-lsb-sdk image again. In resulting image 'uname -v'
will not match UTS_VERSION from
/usr/src/kernel/include/generated/compile.h.

I.e now even kernel-devsrc is pulling information from virtual/kernel,
kernel-devsrc is not rebuilt even if virtual/kernel is
rebuilt. IMHO it should be rebuilt. But is it by design or something
does not work, I cannot tell. In kernel-devsrc there are statements like
'do_install[depends] += "virtual/kernel:do_install"', but whether
those just build ordering statement or whether they should
trigger rebuild if "virtual/kernel:do_install" is newer is not clear.

For now I'll modify stap test patch and call stap with kernel version
check disabled assuming that running kernel and kernel-devsrc always
would be close enough to run without problems. But it would be great if
folks could clarify rebuild dependency rule between virtual/kernel
and kernel-devsrc.

Thanks,
Victor


Ross

On 22 March 2018 at 17:53, Victor Kamensky <kamen...@cisco.com> wrote:

Add runtime test for stap to test basic SystemTap
operations: can compile very basic module and run on
target device.

Signed-off-by: Victor Kamensky <kamen...@cisco.com>
---
 meta/lib/oeqa/runtime/cases/stap.py   | 33 +
 meta/lib/oeqa/runtime/files/hello.stp |  1 +
 2 files changed, 34 insertions(+)
 create mode 100644 meta/lib/oeqa/runtime/cases/stap.py
 create mode 100644 meta/lib/oeqa/runtime/files/hello.stp

diff --git a/meta/lib/oeqa/runtime/cases/stap.py 
b/meta/lib/oeqa/runtime/cases/stap.py
new file mode 100644
index 000..005da71
--- /dev/null
+++ b/meta/lib/oeqa/runtime/cases/stap.py
@@ -0,0 +1,33 @@
+import os
+
+from oeqa.runtime.case import OERuntimeTestCase
+from oeqa.core.decorator.depends import OETestDepends
+from oeqa.core.decorator.oeid import OETestID
+from oeqa.core.decorator.data import skipIfNotFeature
+
+class StapTest(OERuntimeTestCase):
+
+@classmethod
+def setUpClass(cls):
+src = os.path.join(cls.tc.runtime_files_dir, 'hello.stp')
+dst = '/tmp/hello.stp'
+cls.tc.target.copyTo(src, dst)
+
+@classmethod
+def tearDownClass(cls):
+files = '/tmp/hello.stp'
+cls.tc.target.run('rm %s' % files)
+
+@OETestID(1652)
+@skipIfNotFeature('tools-profile',
+  'Test requires tools-profile to be in IMAGE_FEATURES')
+@OETestDepends(['kernelmodule.KernelModuleTest.test_kernel_module'])
+def test_stap(self):
+cmds = [
+'cd /usr/src/kernel && make scripts',
+'cd /lib/modules/* && (if [ ! -L build ]; then ln -s 
/usr/src/kernel build; fi)',
+'stap --disable-cache /tmp/hello.stp'
+]
+for cmd in cmds:
+status, output = self.target.run(cmd, 900)
+self.assertEqual(status, 0, msg='\n'.join([cmd, output]))
dif

Re: [OE-core] [PATCH] qemu: actually use the aarch64 patch

[Victor Kamensky]

On Wed, 28 Mar 2018, Ross Burton wrote:


Signed-off-by: Ross Burton 
---
meta/recipes-devtools/qemu/qemu_2.11.1.bb | 1 +
1 file changed, 1 insertion(+)

diff --git a/meta/recipes-devtools/qemu/qemu_2.11.1.bb 
b/meta/recipes-devtools/qemu/qemu_2.11.1.bb
index 9095c5f..f4b7d69fcad 100644
--- a/meta/recipes-devtools/qemu/qemu_2.11.1.bb
+++ b/meta/recipes-devtools/qemu/qemu_2.11.1.bb
@@ -21,6 +21,7 @@ SRC_URI = 
"http://wiki.qemu-project.org/download/${BP}.tar.bz2 \
   file://apic-fixup-fallthrough-to-PIC.patch \
   file://linux-user-Fix-webkitgtk-hangs-on-32-bit-x86-target.patch \
   file://memfd.patch \
+   
file://0001-arm-translate-a64-treat-DISAS_UPDATE-as-variant-of-D.patch \
   "
UPSTREAM_CHECK_REGEX = "qemu-(?P\d+\..*)\.tar"


Acked and tested with the latest tree and above patch
on machine where the issue is reproducable.

Thank you for taking care of this.

- Victor


--
2.11.0

--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core


--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH] qemu: fix qemuarm64 intermediate kernel hang in raid6_select_algo func

[Victor Kamensky]

On Mon, 26 Mar 2018, Richard Purdie wrote:


On Mon, 2018-03-26 at 11:31 +0100, Burton, Ross wrote:

An early winner of Patch of the Day in my books.


Indeed, getting this one fixed is great as it has been messing up a lot
of builds, thanks!


Thank you guys.

FYI and it is a quite emabarissing: I forgot to do "git add" for .bb
file to update SRC_URI with the patch that I tested. So now it is
really has no effect :(. I will send catch up patch with SRC_URI
update (one that I really tested) as soon as I finish retesting it
on top of latest tree. Sorry about that. I just noticed that while I
reviewed lastest today pulled tree.

Thanks,
Victor


Cheers,

Richard


--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH] qemu: fix qemuarm64 intermediate kernel hang in raid6_select_algo func

[Victor Kamensky]

Backport fix from qemu mainline for intermediate qemuarm64 hang
issue. Root caused in OE environment, issue with aarch64 qemu
logic of executing instructions that reenabe interrupts. See patch
commit message for more details.

Upstream-Status: Backport
Signed-off-by: Victor Kamensky <kamen...@cisco.com>
---
 ...te-a64-treat-DISAS_UPDATE-as-variant-of-D.patch | 67 ++
 1 file changed, 67 insertions(+)
 create mode 100644 
meta/recipes-devtools/qemu/qemu/0001-arm-translate-a64-treat-DISAS_UPDATE-as-variant-of-D.patch

diff --git 
a/meta/recipes-devtools/qemu/qemu/0001-arm-translate-a64-treat-DISAS_UPDATE-as-variant-of-D.patch
 
b/meta/recipes-devtools/qemu/qemu/0001-arm-translate-a64-treat-DISAS_UPDATE-as-variant-of-D.patch
new file mode 100644
index 000..f90cae6
--- /dev/null
+++ 
b/meta/recipes-devtools/qemu/qemu/0001-arm-translate-a64-treat-DISAS_UPDATE-as-variant-of-D.patch
@@ -0,0 +1,67 @@
+From a75a52d62418dafe462be4fe30485501d1010bb9 Mon Sep 17 00:00:00 2001
+From: Victor Kamensky <kamen...@cisco.com>
+Date: Fri, 23 Mar 2018 18:26:45 +
+Subject: [PATCH] arm/translate-a64: treat DISAS_UPDATE as variant of
+ DISAS_EXIT
+MIME-Version: 1.0
+Content-Type: text/plain; charset=UTF-8
+Content-Transfer-Encoding: 8bit
+
+In OE project 4.15 linux kernel boot hang was observed under
+single cpu aarch64 qemu. Kernel code was in a loop waiting for
+vtimer arrival, spinning in TC generated blocks, while interrupt
+was pending unprocessed. This happened because when qemu tried to
+handle vtimer interrupt target had interrupts disabled, as
+result flag indicating TCG exit, cpu->icount_decr.u16.high,
+was cleared but arm_cpu_exec_interrupt function did not call
+arm_cpu_do_interrupt to process interrupt. Later when target
+reenabled interrupts, it happened without exit into main loop, so
+following code that waited for result of interrupt execution
+run in infinite loop.
+
+To solve the problem instructions that operate on CPU sys state
+(i.e enable/disable interrupt), and marked as DISAS_UPDATE,
+should be considered as DISAS_EXIT variant, and should be
+forced to exit back to main loop so qemu will have a chance
+processing pending CPU state updates, including pending
+interrupts.
+
+This change brings consistency with how DISAS_UPDATE is treated
+in aarch32 case.
+
+CC: Peter Maydell <peter.mayd...@linaro.org>
+CC: Alex Bennée <alex.ben...@linaro.org>
+CC: qemu-sta...@nongnu.org
+Suggested-by: Peter Maydell <peter.mayd...@linaro.org>
+Signed-off-by: Victor Kamensky <kamen...@cisco.com>
+Reviewed-by: Richard Henderson <richard.hender...@linaro.org>
+Message-id: 1521526368-1996-1-git-send-email-kamen...@cisco.com
+Signed-off-by: Peter Maydell <peter.mayd...@linaro.org>
+Upstream-Status: Backport
+---
+ target/arm/translate-a64.c | 6 +++---
+ 1 file changed, 3 insertions(+), 3 deletions(-)
+
+diff --git a/target/arm/translate-a64.c b/target/arm/translate-a64.c
+index 31ff047..327513e 100644
+--- a/target/arm/translate-a64.c
 b/target/arm/translate-a64.c
+@@ -13378,12 +13378,12 @@ static void aarch64_tr_tb_stop(DisasContextBase 
*dcbase, CPUState *cpu)
+ case DISAS_UPDATE:
+ gen_a64_set_pc_im(dc->pc);
+ /* fall through */
+-case DISAS_JUMP:
+-tcg_gen_lookup_and_goto_ptr();
+-break;
+ case DISAS_EXIT:
+ tcg_gen_exit_tb(0);
+ break;
++case DISAS_JUMP:
++tcg_gen_lookup_and_goto_ptr();
++break;
+ case DISAS_NORETURN:
+ case DISAS_SWI:
+ break;
+-- 
+2.7.4
+
-- 
2.7.4

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] pseudo: host user contamination

[Victor Kamensky]

On Sat, 24 Mar 2018, Andre McCurdy wrote:


On Sat, Mar 24, 2018 at 5:09 PM, Victor Kamensky <kamen...@cisco.com> wrote:

On Sat, 24 Mar 2018, Burton, Ross wrote:

On 24 March 2018 at 20:12, Victor Kamensky <kamen...@cisco.com> wrote:


Here is another crazy idea how to deal with it, just
brainstorming what options are on the table: disable
renameat2 with help of seccomp and force coreutils to
use other calls. Something along the lines that were
suggested with intercept of syscall function call, but
let kernel to do interception work.


Wow, that's impressively magic.  Does this depend on kernel options or
specific recent versions?


Yeah, it's impressive but perhaps overkill for this situation.

Having the kernel run a BPF script on every syscall is going to have a
much bigger performance impact than intercepting one specific libc
function in user space.


I don't think we should worry about overhead in pseudo case.


Also, AFAIK, seccomp can't be nested - so building within an
environment which has already been secured with seccomp (e.g. recent
versions of docker?) might be a problem if pseudo starts to rely on
seccomp too.


Above is true. It was on my mind.

Note I have no problem whatsoever if you can intercept syscall
function correctly. Function intercepting way is definitely more
aligned with what pseudo does. I was just listing other
possible options.

But please note syscall function takes a
variable number of arguments and call another variable
number of argument function, real syscall implementation, in
general, cannot be done. One would need to have complimentary
vsyscall function taking va_list. I.e like printf and vprintf.

Please see http://c-faq.com/varargs/handoff.html

But maybe something special can be done for syscall case.
Disclaimer: I did not read full thread, maybe you already
discussed this.

Thanks,
Victor
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] pseudo: host user contamination

[Victor Kamensky]

On Sat, 24 Mar 2018, Burton, Ross wrote:


On 24 March 2018 at 20:12, Victor Kamensky <kamen...@cisco.com> wrote:

Here is another crazy idea how to deal with it, just
brainstorming what options are on the table: disable
renameat2 with help of seccomp and force coreutils to
use other calls. Something along the lines that were
suggested with intercept of syscall function call, but
let kernel to do interception work.


Wow, that's impressively magic.  Does this depend on kernel options or
specific recent versions?


Not very recent, but relatively mordern. As far as I read
kernel code seccomp syscall BPF filtering [1] was introduced
in 2012 in 3.5 kernel by chromium project guys.

It is controlled by CONFIG_SECCOMP_FILTER which depends on
HAVE_ARCH_SECCOMP_FILTER that all major CPU architectures
do support by now. And I think CONFIG_SECCOMP_FILTER should
be set for all major cases - AFAIK chrome browser uses it
as one of its sandboxing mechanisms.

But you are right, if any code would use it, it needs to
check whether usable seccomp syscall filtering is present
on the system.

[1] 
https://github.com/torvalds/linux/blob/master/Documentation/userspace-api/seccomp_filter.rst

Thanks,
Victor


Ross


--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] pseudo: host user contamination

[Victor Kamensky]

Here is another crazy idea how to deal with it, just
brainstorming what options are on the table: disable
renameat2 with help of seccomp and force coreutils to
use other calls. Something along the lines that were
suggested with intercept of syscall function call, but
let kernel to do interception work.

Here is tiny example based on my todays learning or
seccomp and eBPF, it shows how on my FC27 filtering out
renameat2 forces coreutils mv do use other calls to do the job.

[kamensky@coreos-lnx2 bpf]$ cat filterout_renameat2.c
#include 
#include 
#include 
#include 
#include 
#include 

#define syscall_nr (offsetof(struct seccomp_data, nr))

struct sock_filter filterout_renameat2[] = {
BPF_STMT(BPF_LD+BPF_W+BPF_ABS, syscall_nr),
BPF_JUMP(BPF_JMP+BPF_JEQ+BPF_K, __NR_renameat2, 0, 1),
BPF_STMT(BPF_RET+BPF_K, SECCOMP_RET_ERRNO + ENOSYS),
BPF_STMT(BPF_RET+BPF_K, SECCOMP_RET_ALLOW)
};

struct sock_fprog filterout_renameat2_prog = {
.len = (unsigned short)(sizeof(filterout_renameat2) /
sizeof(filterout_renameat2[0])),
.filter = filterout_renameat2,
};

int disable_renameat2_syscall (void)
{
int err;
err = prctl(PR_SET_NO_NEW_PRIVS, 1, 0, 0, 0);
if (!err) {
err = prctl(PR_SET_SECCOMP, SECCOMP_MODE_FILTER,
_renameat2_prog);
}

return err;
}
[kamensky@coreos-lnx2 bpf]$ cat norenameat2.c
#include 
#include 

int disable_renameat2_syscall (void);

int main(int argc, char **argv)
{
int err = 0;

err = disable_renameat2_syscall();
if(err) {
perror("disable_renameat2_syscall");
}

execvp (argv[1], [1]);
return 0;
}
[kamensky@coreos-lnx2 bpf]$ gcc -o norenameat2 norenameat2.c filterout_renameat2.c 
[kamensky@coreos-lnx2 bpf]$ mkdir foo

[kamensky@coreos-lnx2 bpf]$ strace -o ./trace.mv.txt -f mv foo bar
[kamensky@coreos-lnx2 bpf]$ grep rename ./trace.mv.txt
2218  renameat2(AT_FDCWD, "foo", AT_FDCWD, "bar", 0) = 0
[kamensky@coreos-lnx2 bpf]$ rm -r -f bar
[kamensky@coreos-lnx2 bpf]$ mkdir foo
[kamensky@coreos-lnx2 bpf]$ strace -o ./trace.norenameat2.mv.txt -f 
./norenameat2 mv foo bar
[kamensky@coreos-lnx2 bpf]$ grep rename ./trace.norenameat2.mv.txt
2228  execve("./norenameat2", ["./norenameat2", "mv", "foo", "bar"], 
0x7ffd16d930e0 /* 37 vars */) = 0
2228  renameat2(AT_FDCWD, "foo", AT_FDCWD, "bar", 0) = -1 ENOSYS (Function not 
implemented)
2228  renameat(AT_FDCWD, "foo", AT_FDCWD, "bar") = 0
[kamensky@coreos-lnx2 bpf]$

Thanks,
Victor

On Sat, 24 Mar 2018, Seebs wrote:


On Sat, 24 Mar 2018 12:42:45 -0700
Andre McCurdy  wrote:


Right. The musl example is to show how it's possible to transparently
intercept and pass on any call to the syscall() ABI without
interpreting anything.


Yes, if you don't need to interpret things, and aren't making
additional other unrelated system calls after doing so.


Those details are all taken care of within the libc implementation of
syscall(). It's not something we need to care about at all in a
wrapper for it.


I don't think that's correct.

musl's call sequence:
real_syscall() // sets a3
return

pseudo's call sequence:
various_setup()
real_syscall() // sets a3
other system calls // also set a3
return

In the case where pseudo is actually *disabled*, we just return
right away after the real call. In every other case, we're making
other calls some of which imply system calls, and those system calls
could potentially overwrite things that the libc implementation of
syscall took care of. (Mutex and signal mask operations.)

So for that to work, I would in principle have to stash the value
stored in, for instance, "a3", wait until after the other system calls,
and then restore it. Unless *only* syscall() itself actually sets
that register, and other system calls don't, and nothing else is
using it either.

-s
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core


--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH] oeqa/runtime/stap.py: add runtime test for systemtap

[Victor Kamensky]

Add runtime test for stap to test basic SystemTap
operations: can compile very basic module and run on
target device.

Signed-off-by: Victor Kamensky <kamen...@cisco.com>
---
 meta/lib/oeqa/runtime/cases/stap.py   | 33 +
 meta/lib/oeqa/runtime/files/hello.stp |  1 +
 2 files changed, 34 insertions(+)
 create mode 100644 meta/lib/oeqa/runtime/cases/stap.py
 create mode 100644 meta/lib/oeqa/runtime/files/hello.stp

diff --git a/meta/lib/oeqa/runtime/cases/stap.py 
b/meta/lib/oeqa/runtime/cases/stap.py
new file mode 100644
index 000..005da71
--- /dev/null
+++ b/meta/lib/oeqa/runtime/cases/stap.py
@@ -0,0 +1,33 @@
+import os
+
+from oeqa.runtime.case import OERuntimeTestCase
+from oeqa.core.decorator.depends import OETestDepends
+from oeqa.core.decorator.oeid import OETestID
+from oeqa.core.decorator.data import skipIfNotFeature
+
+class StapTest(OERuntimeTestCase):
+
+@classmethod
+def setUpClass(cls):
+src = os.path.join(cls.tc.runtime_files_dir, 'hello.stp')
+dst = '/tmp/hello.stp'
+cls.tc.target.copyTo(src, dst)
+
+@classmethod
+def tearDownClass(cls):
+files = '/tmp/hello.stp'
+cls.tc.target.run('rm %s' % files)
+
+@OETestID(1652)
+@skipIfNotFeature('tools-profile',
+  'Test requires tools-profile to be in IMAGE_FEATURES')
+@OETestDepends(['kernelmodule.KernelModuleTest.test_kernel_module'])
+def test_stap(self):
+cmds = [
+'cd /usr/src/kernel && make scripts',
+'cd /lib/modules/* && (if [ ! -L build ]; then ln -s 
/usr/src/kernel build; fi)',
+'stap --disable-cache /tmp/hello.stp'
+]
+for cmd in cmds:
+status, output = self.target.run(cmd, 900)
+self.assertEqual(status, 0, msg='\n'.join([cmd, output]))
diff --git a/meta/lib/oeqa/runtime/files/hello.stp 
b/meta/lib/oeqa/runtime/files/hello.stp
new file mode 100644
index 000..3677147
--- /dev/null
+++ b/meta/lib/oeqa/runtime/files/hello.stp
@@ -0,0 +1 @@
+probe oneshot { println("hello world") }
-- 
2.7.4

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH] systemtap runtime test, ping for outstanding systemtap fixes

[Victor Kamensky]

Hi Folks,

Based on my conversation with Richard during conference please find
patch that does native SystemTap stap basic test. Note it is for native
stap use. It is not SystemTap cross compilation test that Ee Peng
is trying to cover in [1]. Actually, I wrote this small test before
I saw message from Ee Peng for crosstap test, but I did not have
time to post it.

Note native SystemTap stap run on qemu machine only feasible on x86
qemu. IMO it would run too long on qemu target with other CPU types.
Test should be executed against something like core-image-lsb-sdk
image, otherwise it would be skipped.

Also, could someone, if possible, please merge outstanding fixes for
systemtap [2] and [3]. Khem did ack on both a while back. Currently
with 4.15 kernel in openembedded-core systemtap in native or cross
mode is broken. Test I am posting would catch current breakage. When
[3] is applied test would pass.

[1] 
http://lists.openembedded.org/pipermail/openembedded-core/2018-March/148972.html

[2] 
http://lists.openembedded.org/pipermail/openembedded-core/2018-March/148348.html

[3] 
http://lists.openembedded.org/pipermail/openembedded-core/2018-March/148349.html

Thanks,
Victor

Victor Kamensky (1):
  oeqa/runtime/stap.py: add runtime test for systemtap

 meta/lib/oeqa/runtime/cases/stap.py   | 33 +
 meta/lib/oeqa/runtime/files/hello.stp |  1 +
 2 files changed, 34 insertions(+)
 create mode 100644 meta/lib/oeqa/runtime/cases/stap.py
 create mode 100644 meta/lib/oeqa/runtime/files/hello.stp

-- 
2.7.4

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] Need arm64/qemu help

[Victor Kamensky]

On Mon, 19 Mar 2018, Richard Henderson wrote:


On 03/20/2018 07:24 AM, Victor Kamensky wrote:

target/arm/translate-a64.c:handle_sys() is setting
   s->base.is_jmp = DISAS_UPDATE;
which it thinks will end the TB, specifically because system
register writes might do things like unmask interrupts or
otherwise require main loop processing.

The changes that prompted b29fd33db578dec stopped this working.
I suspect what we want is for the case DISAS_UPDATE in
aarch64_tr_tb_stop() to fall through into DISAS_EXIT, not
DISAS_JUMP. (The AArch32 code gets this right, amazingly.)


Peter, thank you. I can confirm that change you suggested
like one below, boots fine.

diff --git a/target/arm/translate-a64.c b/target/arm/translate-a64.c
index 625ef2d..c381091 100644
--- a/target/arm/translate-a64.c
+++ b/target/arm/translate-a64.c
@@ -11384,12 +11384,12 @@ static void aarch64_tr_tb_stop(DisasContextBase
*dcbase, CPUState *cpu)
 case DISAS_UPDATE:
 gen_a64_set_pc_im(dc->pc);
 /* fall through */
-    case DISAS_JUMP:
-    tcg_gen_lookup_and_goto_ptr();
-    break;
 case DISAS_EXIT:
 tcg_gen_exit_tb(0);
 break;
+    case DISAS_JUMP:
+    tcg_gen_lookup_and_goto_ptr();
+    break;
 case DISAS_NORETURN:
 case DISAS_SWI:
 break;


Alex and I just had a long chat about this, and I think this is the right 
solution.

(1) It brings aa64 into alignment with aa32 wrt the meaning of these
enumerators, (2) The only use of DISAS_UPDATE is there in handle_sys, so we're
not changing the behaviour of any other insns, (3) The most frequent writes to
system registers -- NZCV and FPCR -- are already special cased so that they
won't exit the TB.

Reviewed-by: Richard Henderson <richard.hender...@linaro.org>


Thanks, Richard. So how do we go about committing it in qemu tree?
Should I submit the patch or Alex or Peter will do it?

Once fix is in qemu tree I will pick it and submit patch for OE tree
on top of current qemu-2.11.1.

Thanks,
Victor



r~
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] Need arm64/qemu help

[Victor Kamensky]

On Mon, 19 Mar 2018, Alex Bennée wrote:



Peter Maydell <peter.mayd...@linaro.org> writes:


On 19 March 2018 at 17:46, Victor Kamensky <kamen...@cisco.com> wrote:

In v2.11.1 of qemu, that we use, we already have
b29fd33db578decacd14f34933b29aece3e7c25e. Previous testing
and collected log was done with it present.

But my understanding that eret would happen when target exits
an interrupt, here I don't think it enters one.

Consider that target explicitely disables interrupts and while it is
disabled, arm_cpu_exec_interrupt function calls arm_excp_unmasked
and it returns false, so arm_cpu_do_interrupt is not called. Main
loop resume execution, and one of the block explicitely
reenables interrupt and sequence continues without ever returning to
main loop.

For example, if I apply below patch, it boots fine. But I am not sure
in what other places similar thing is needed, and whether below
is complete and correct:

diff --git a/target/arm/helper.c b/target/arm/helper.c
index 91a9300..19128c5 100644
--- a/target/arm/helper.c
+++ b/target/arm/helper.c
@@ -2948,6 +2948,14 @@ static CPAccessResult aa64_daif_access(CPUARMState
*env, const ARMCPRegInfo *ri,
 static void aa64_daif_write(CPUARMState *env, const ARMCPRegInfo *ri,
 uint64_t value)
 {
+if (env->daif & ~(value & PSTATE_DAIF)) {
+/* reenabling interrupts */
+CPUState *cs = CPU(arm_env_get_cpu(env));
+if (cs->interrupt_request) {
+/* there is pending one, let's drop back into main loop */
+cs->icount_decr.u16.high = -1;
+}
+}
 env->daif = value & PSTATE_DAIF;
 }


target/arm/translate-a64.c:handle_sys() is setting
s->base.is_jmp = DISAS_UPDATE;
which it thinks will end the TB, specifically because system
register writes might do things like unmask interrupts or
otherwise require main loop processing.


For the DAIFclear and eret paths we set DISAS_EXIT. What is the
handle_sys path that should be doing this? Is this a direct setting of
DAIF?


Yes, the one that translated into aa64_daif_write help invocation, ie
something like:

'msr daif, x25'

the reason why I went in my experiment after aa64_daif_write function,
that I saw it was hitting daif watchpoint last, clearing it before
system hangged.

Here is backtrace before system stuck. After backtrace,
first entry is interrupt_requested, followed by daif, and then by
cp15.hcr_el2, and cp15.scr_el3

Old value = 128
New value = 0
aa64_daif_write (env=0x18c8430, ri=0x18f07d0, value=0)
at 
/wd6/oe/20180311/build/tmp-glibc/work/x86_64-linux/qemu-native/2.11.1-r0/qemu-2.11.1/target/arm/helper.c:2952
2952}
#0  aa64_daif_write (env=0x18c8430, ri=0x18f07d0, value=0)
at 
/wd6/oe/20180311/build/tmp-glibc/work/x86_64-linux/qemu-native/2.11.1-r0/qemu-2.11.1/target/arm/helper.c:2952
#1  0x005c8f43 in helper_set_cp_reg64 (env=0x18c8430, rip=0x18f07d0, 
value=0)
at 
/wd6/oe/20180311/build/tmp-glibc/work/x86_64-linux/qemu-native/2.11.1-r0/qemu-2.11.1/target/arm/op_helper.c:842
#2  0x7fffec05cec7 in code_gen_buffer ()
#3  0x0048aee9 in cpu_tb_exec (cpu=0x18c0190, itb=0x7fffec0393c0 
<code_gen_buffer+164758>)
at 
/wd6/oe/20180311/build/tmp-glibc/work/x86_64-linux/qemu-native/2.11.1-r0/qemu-2.11.1/accel/tcg/cpu-exec.c:167
#4  0x0048bd82 in cpu_loop_exec_tb (cpu=0x18c0190, tb=0x7fffec0393c0 
<code_gen_buffer+164758>, last_tb=0x7fffec00faf8,
tb_exit=0x7fffec00faf0) at 
/wd6/oe/20180311/build/tmp-glibc/work/x86_64-linux/qemu-native/2.11.1-r0/qemu-2.11.1/accel/tcg/cpu-exec.c:627
#5  0x0048c091 in cpu_exec (cpu=0x18c0190)
at 
/wd6/oe/20180311/build/tmp-glibc/work/x86_64-linux/qemu-native/2.11.1-r0/qemu-2.11.1/accel/tcg/cpu-exec.c:736
#6  0x0044a883 in tcg_cpu_exec (cpu=0x18c0190)
at 
/wd6/oe/20180311/build/tmp-glibc/work/x86_64-linux/qemu-native/2.11.1-r0/qemu-2.11.1/cpus.c:1270
#7  0x0044ad82 in qemu_tcg_cpu_thread_fn (arg=0x18c0190)
at 
/wd6/oe/20180311/build/tmp-glibc/work/x86_64-linux/qemu-native/2.11.1-r0/qemu-2.11.1/cpus.c:1475
#8  0x779616ba in start_thread (arg=0x7fffec010700) at 
pthread_create.c:333
#9  0x759bc41d in clone () at 
../sysdeps/unix/sysv/linux/x86_64/clone.S:109
$7778 = 0x2
$7779 = 0x0
$7780 = 0x0
$7781 = 0x0

Note IMO dealing with aa64_daif_write may not be sufficient, because
besides daif unmasked interrupt function also checks cp15.hcr_el2, and 
cp15.scr_el3. Those could be a reason why interrupt is masked and

they could change too and require exit into main loop, if interrupt is
pending.

Thanks,
Victor



The changes that prompted b29fd33db578dec stopped this working.
I suspect what we want is for the case DISAS_UPDATE in
aarch64_tr_tb_stop() to fall through into DISAS_EXIT, not
DISAS_JUMP. (The AArch32 code gets this right, amazingly.)

thanks
-- PMM



--
Alex Bennée
-- 
___
Openembedded-core mailing

Re: [OE-core] Need arm64/qemu help

[Victor Kamensky]

On Mon, 19 Mar 2018, Peter Maydell wrote:


On 19 March 2018 at 17:46, Victor Kamensky <kamen...@cisco.com> wrote:

In v2.11.1 of qemu, that we use, we already have
b29fd33db578decacd14f34933b29aece3e7c25e. Previous testing
and collected log was done with it present.

But my understanding that eret would happen when target exits
an interrupt, here I don't think it enters one.

Consider that target explicitely disables interrupts and while it is
disabled, arm_cpu_exec_interrupt function calls arm_excp_unmasked
and it returns false, so arm_cpu_do_interrupt is not called. Main
loop resume execution, and one of the block explicitely
reenables interrupt and sequence continues without ever returning to
main loop.

For example, if I apply below patch, it boots fine. But I am not sure
in what other places similar thing is needed, and whether below
is complete and correct:

diff --git a/target/arm/helper.c b/target/arm/helper.c
index 91a9300..19128c5 100644
--- a/target/arm/helper.c
+++ b/target/arm/helper.c
@@ -2948,6 +2948,14 @@ static CPAccessResult aa64_daif_access(CPUARMState
*env, const ARMCPRegInfo *ri,
 static void aa64_daif_write(CPUARMState *env, const ARMCPRegInfo *ri,
 uint64_t value)
 {
+if (env->daif & ~(value & PSTATE_DAIF)) {
+/* reenabling interrupts */
+CPUState *cs = CPU(arm_env_get_cpu(env));
+if (cs->interrupt_request) {
+/* there is pending one, let's drop back into main loop */
+cs->icount_decr.u16.high = -1;
+}
+}
 env->daif = value & PSTATE_DAIF;
 }


target/arm/translate-a64.c:handle_sys() is setting
   s->base.is_jmp = DISAS_UPDATE;
which it thinks will end the TB, specifically because system
register writes might do things like unmask interrupts or
otherwise require main loop processing.

The changes that prompted b29fd33db578dec stopped this working.
I suspect what we want is for the case DISAS_UPDATE in
aarch64_tr_tb_stop() to fall through into DISAS_EXIT, not
DISAS_JUMP. (The AArch32 code gets this right, amazingly.)


Peter, thank you. I can confirm that change you suggested
like one below, boots fine.

diff --git a/target/arm/translate-a64.c b/target/arm/translate-a64.c
index 625ef2d..c381091 100644
--- a/target/arm/translate-a64.c
+++ b/target/arm/translate-a64.c
@@ -11384,12 +11384,12 @@ static void aarch64_tr_tb_stop(DisasContextBase 
*dcbase, CPUState *cpu)

 case DISAS_UPDATE:
 gen_a64_set_pc_im(dc->pc);
 /* fall through */
-case DISAS_JUMP:
-tcg_gen_lookup_and_goto_ptr();
-break;
 case DISAS_EXIT:
 tcg_gen_exit_tb(0);
 break;
+case DISAS_JUMP:
+tcg_gen_lookup_and_goto_ptr();
+break;
 case DISAS_NORETURN:
 case DISAS_SWI:
 break;

- Victor


thanks
-- PMM


--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] Need arm64/qemu help

[Victor Kamensky]

On Mon, 19 Mar 2018, Alex Bennée wrote:



Victor Kamensky <kamen...@cisco.com> writes:


On Sun, 18 Mar 2018, Alex Bennée wrote:



Victor Kamensky <kamen...@cisco.com> writes:


Hi Folks,

I am back from ELC and I am looking at this issue again.

Alex Bennée from Linaro QEMU team reported that he tried to
reproduce the issue, even he went and build the same OE images -
but he could not reproduce it. Alex, thank you for the effort.

So since it is reprodicible on my machine I kept digging
myself. In no way I am qemu expert, but I think now I
understand what is happening. Long story with my debug notes
is below, but here is executive summery:

Linux kernel loop waiting for jiffies to move on, while
calling yield instruction, in our case aarch64 target runs
under one CPU configuration, and after Alex's commit "c22edfe
target-arm: don't generate WFE/YIELD calls for MTTCG" qemu
logic of handling yield instruction changed in such way
that it is treated as simple nop. But since it is single CPU
configuration, we have qemu looping in generated code
forver without existing loop to process pending vtimer
interrupt in order to move jiffies forward. c22edfe implies
that it is not parallel CPU case, but in our case even if
we have single CPU target parallel execution is still turned
on.


Hmm well my gut guess was right but I'm not sure how it is hanging.

First of all could you confirm it hangs/works with the invocation:

 --accel tcg,thread=single -smp 2


'--accel tcg,thread=single -smp 2' works fine. '--accel tcg,thread=single'
works fine too.


The parallel_cpus variable only refers to the fact MTTCG is enabled or
not. You can still model -smp > 1 systems without MTTCG but in this case
each vCPU will be run round-robin style from a single thread.

What is troubling is you are not seeing the timer IRQ fire. The whole
point of the cpu_loop_exit(cs) is to ensure we start again on the next
vCPU. Every time we enter a new block we check to see if an IRQ has
fired and then process it then.


Thank you for guidance. Now I see icount_decr.u16.high check
in start of every generated block. Setting icount_decr.u16.high to -1,
does drop target out of loop, but now I think that the code that acts
on it, cpu_handle_interrupt function, clears icount_decr.u16.high,
but it cannot really serve the interrupt because it is disabled by
target, as result code comes back with interupt_request set, and
icount_decr.u16.high cleared, and resume blocks execution which
hangs in sequence we discussed. But as before I could be missing
something again.

Please see more details and new gdb session log in line.


Revert of c22edfe fixes the issue, image boots OK. Booting
with more than one CPU "-smp 2" boots fine too. And possibly
could work as solid workaround for us. But how to fix the
issue without revert, while moving forward, I don't know.
I hope Alex and Linaro QEMU folks can come up with something.

Now, long debugging notes with my comments:

Kernel Hang Backtrace
-

Looking at kernel hang under gdb attached to qemu (runqemu
qemuparams="-s -S" bootparams="nokaslr"):

(gdb) bt
#0  vectors () at /usr/src/kernel/arch/arm64/kernel/entry.S:413
#1  0xff80089a3758 in raid6_choose_gen (disks=, 
dptrs=) at /usr/src/kernel/lib/raid6/algos.c:190
#2  raid6_select_algo () at /usr/src/kernel/lib/raid6/algos.c:253
#3  0xff8008083bdc in do_one_initcall (fn=0xff80089a35c8 
) at /usr/src/kernel/init/main.c:832
#4  0xff8008970e80 in do_initcall_level (level=) at 
/usr/src/kernel/init/main.c:898
#5  do_initcalls () at /usr/src/kernel/init/main.c:906
#6  do_basic_setup () at /usr/src/kernel/init/main.c:924
#7  kernel_init_freeable () at /usr/src/kernel/init/main.c:1073
#8  0xff80087afd90 in kernel_init (unused=) at 
/usr/src/kernel/init/main.c:999
#9  0xff800808517c in ret_from_fork () at 
/usr/src/kernel/arch/arm64/kernel/entry.S:1154
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
(gdb) f 1
#1  0xff80089a3758 in raid6_choose_gen (disks=, 
dptrs=) at /usr/src/kernel/lib/raid6/algos.c:190
190 continue;
(gdb) x /12i $pc - 12
   0xff80089a374c <raid6_select_algo+388>:cbz x0, 0xff80089a37fc 
<raid6_select_algo+564>
   0xff80089a3750 <raid6_select_algo+392>:mov w0, #0x1  
  // #1
   0xff80089a3754 <raid6_select_algo+396>:bl  0xff80080d07c8 

=> 0xff80089a3758 <raid6_select_algo+400>: ldr x0, [x23, #2688]
   0xff80089a375c <raid6_select_algo+404>:ldr x5, [x23, #2688]
   0xff80089a3760 <raid6_select_algo+408>:cmp x0, x5
   0xff80089a3764 <raid6_select_algo+412>:b.ne0xff80089a3770 
<raid6_select_algo+424>  // b.any
   0xff80089a3768 <raid6_select_algo+416>:yield
   0xff80089a376c <raid6_select_algo+420>

Re: [OE-core] Need arm64/qemu help

[Victor Kamensky]

On Sun, 18 Mar 2018, Alex Bennée wrote:



Victor Kamensky <kamen...@cisco.com> writes:


Hi Folks,

I am back from ELC and I am looking at this issue again.

Alex Bennée from Linaro QEMU team reported that he tried to
reproduce the issue, even he went and build the same OE images -
but he could not reproduce it. Alex, thank you for the effort.

So since it is reprodicible on my machine I kept digging
myself. In no way I am qemu expert, but I think now I
understand what is happening. Long story with my debug notes
is below, but here is executive summery:

Linux kernel loop waiting for jiffies to move on, while
calling yield instruction, in our case aarch64 target runs
under one CPU configuration, and after Alex's commit "c22edfe
target-arm: don't generate WFE/YIELD calls for MTTCG" qemu
logic of handling yield instruction changed in such way
that it is treated as simple nop. But since it is single CPU
configuration, we have qemu looping in generated code
forver without existing loop to process pending vtimer
interrupt in order to move jiffies forward. c22edfe implies
that it is not parallel CPU case, but in our case even if
we have single CPU target parallel execution is still turned
on.


Hmm well my gut guess was right but I'm not sure how it is hanging.

First of all could you confirm it hangs/works with the invocation:

 --accel tcg,thread=single -smp 2


'--accel tcg,thread=single -smp 2' works fine. '--accel tcg,thread=single'
works fine too.


The parallel_cpus variable only refers to the fact MTTCG is enabled or
not. You can still model -smp > 1 systems without MTTCG but in this case
each vCPU will be run round-robin style from a single thread.

What is troubling is you are not seeing the timer IRQ fire. The whole
point of the cpu_loop_exit(cs) is to ensure we start again on the next
vCPU. Every time we enter a new block we check to see if an IRQ has
fired and then process it then.


Thank you for guidance. Now I see icount_decr.u16.high check
in start of every generated block. Setting icount_decr.u16.high to -1,
does drop target out of loop, but now I think that the code that acts
on it, cpu_handle_interrupt function, clears icount_decr.u16.high,
but it cannot really serve the interrupt because it is disabled by
target, as result code comes back with interupt_request set, and
icount_decr.u16.high cleared, and resume blocks execution which
hangs in sequence we discussed. But as before I could be missing
something again.

Please see more details and new gdb session log in line.


Revert of c22edfe fixes the issue, image boots OK. Booting
with more than one CPU "-smp 2" boots fine too. And possibly
could work as solid workaround for us. But how to fix the
issue without revert, while moving forward, I don't know.
I hope Alex and Linaro QEMU folks can come up with something.

Now, long debugging notes with my comments:

Kernel Hang Backtrace
-

Looking at kernel hang under gdb attached to qemu (runqemu
qemuparams="-s -S" bootparams="nokaslr"):

(gdb) bt
#0  vectors () at /usr/src/kernel/arch/arm64/kernel/entry.S:413
#1  0xff80089a3758 in raid6_choose_gen (disks=, 
dptrs=) at /usr/src/kernel/lib/raid6/algos.c:190
#2  raid6_select_algo () at /usr/src/kernel/lib/raid6/algos.c:253
#3  0xff8008083bdc in do_one_initcall (fn=0xff80089a35c8 
) at /usr/src/kernel/init/main.c:832
#4  0xff8008970e80 in do_initcall_level (level=) at 
/usr/src/kernel/init/main.c:898
#5  do_initcalls () at /usr/src/kernel/init/main.c:906
#6  do_basic_setup () at /usr/src/kernel/init/main.c:924
#7  kernel_init_freeable () at /usr/src/kernel/init/main.c:1073
#8  0xff80087afd90 in kernel_init (unused=) at 
/usr/src/kernel/init/main.c:999
#9  0xff800808517c in ret_from_fork () at 
/usr/src/kernel/arch/arm64/kernel/entry.S:1154
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
(gdb) f 1
#1  0xff80089a3758 in raid6_choose_gen (disks=, 
dptrs=) at /usr/src/kernel/lib/raid6/algos.c:190
190 continue;
(gdb) x /12i $pc - 12
   0xff80089a374c <raid6_select_algo+388>:cbz x0, 0xff80089a37fc 
<raid6_select_algo+564>
   0xff80089a3750 <raid6_select_algo+392>:mov w0, #0x1  
  // #1
   0xff80089a3754 <raid6_select_algo+396>:bl  0xff80080d07c8 

=> 0xff80089a3758 <raid6_select_algo+400>: ldr x0, [x23, #2688]
   0xff80089a375c <raid6_select_algo+404>:ldr x5, [x23, #2688]
   0xff80089a3760 <raid6_select_algo+408>:cmp x0, x5
   0xff80089a3764 <raid6_select_algo+412>:b.ne0xff80089a3770 
<raid6_select_algo+424>  // b.any
   0xff80089a3768 <raid6_select_algo+416>:yield
   0xff80089a376c <raid6_select_algo+420>:b   0xff80089a375c 
<raid6_select_algo+404>
   0xff80089a3770 <raid6_s

Re: [OE-core] Need arm64/qemu help

[Victor Kamensky]

x18a0ef0, errp=0x0)
at 
/wd6/oe/20180311/build/tmp-glibc/work/x86_64-linux/qemu-native/2.11.1-r0/qemu-2.11.1/qom/object.c:1906
#5  0x008c9cfb in object_property_set (obj=0x18c0190, v=0x18e1db0, name=0xb00953 
"realized", errp=0x0)
at 
/wd6/oe/20180311/build/tmp-glibc/work/x86_64-linux/qemu-native/2.11.1-r0/qemu-2.11.1/qom/object.c:1102
#6  0x0083 in object_property_set_qobject (obj=0x18c0190, value=0x18e1d30, 
name=0xb00953 "realized", errp=0x0)
at 
/wd6/oe/20180311/build/tmp-glibc/work/x86_64-linux/qemu-native/2.11.1-r0/qemu-2.11.1/qom/qom-qobject.c:27
#7  0x008c9f90 in object_property_set_bool (obj=0x18c0190, value=true, 
name=0xb00953 "realized", errp=0x0)
at 
/wd6/oe/20180311/build/tmp-glibc/work/x86_64-linux/qemu-native/2.11.1-r0/qemu-2.11.1/qom/object.c:1171
#8  0x005575ec in machvirt_init (machine=0x1880fa0)
at 
/wd6/oe/20180311/build/tmp-glibc/work/x86_64-linux/qemu-native/2.11.1-r0/qemu-2.11.1/hw/arm/virt.c:1405
#9  0x006de738 in machine_run_board_init (machine=0x1880fa0)
at 
/wd6/oe/20180311/build/tmp-glibc/work/x86_64-linux/qemu-native/2.11.1-r0/qemu-2.11.1/hw/core/machine.c:793
#10 0x0063e08b in main (argc=24, argv=0x7fffdb98, 
envp=0x7fffdc60)
at 
/wd6/oe/20180311/build/tmp-glibc/work/x86_64-linux/qemu-native/2.11.1-r0/qemu-2.11.1/vl.c:4753
(gdb) list
1690qemu_cond_init(cpu->halt_cond);
1691 
1692	if (qemu_tcg_mttcg_enabled()) {

1693/* create a thread per vCPU with TCG (MTTCG) */
1694parallel_cpus = true;
1695snprintf(thread_name, VCPU_THREAD_NAME_SIZE, "CPU %d/TCG",
1696 cpu->cpu_index);
1697

Here is I am a bit lost - it looks like logic of running separate thread for CPU
implies parallel_cpus=true, even number of CPUs is one.

Now sure how to fix it that by moving forward. Hopefully Alex and other
Linaro QEMU folks can chip in.

Revert of c22edfe with resolved conflicts
-

Image boots OK if this applied:


From 3322ab069015da3d3c2f80ce714d4fb99b7d3b7f Mon Sep 17 00:00:00 2001

From: Victor Kamensky <kamen...@cisco.com>
Date: Sat, 17 Mar 2018 15:05:39 -0700
Subject: [PATCH] Revert "target-arm: don't generate WFE/YIELD calls for MTTCG"

This reverts commit c22edfebff29f63d793032e4fbd42a035bb73e27.

On single CPU aarch64 target loop with cpu_relax (yield) stuck
in loop forever, while pending vtimer interrupt raised
un-processed by outside loop.

Signed-off-by: Victor Kamensky <kamen...@cisco.com>
---
 target/arm/translate-a64.c |  8 ++--
 target/arm/translate.c | 20 
 2 files changed, 6 insertions(+), 22 deletions(-)

diff --git a/target/arm/translate-a64.c b/target/arm/translate-a64.c
index 625ef2d..b0cdb64 100644
--- a/target/arm/translate-a64.c
+++ b/target/arm/translate-a64.c
@@ -1342,14 +1342,10 @@ static void handle_hint(DisasContext *s, uint32_t insn,
  * spin unnecessarily we would need to do something more involved.
  */
 case 1: /* YIELD */
-if (!(tb_cflags(s->base.tb) & CF_PARALLEL)) {
-s->base.is_jmp = DISAS_YIELD;
-}
+s->base.is_jmp = DISAS_YIELD;
 return;
 case 2: /* WFE */
-if (!(tb_cflags(s->base.tb) & CF_PARALLEL)) {
-s->base.is_jmp = DISAS_WFE;
-}
+s->base.is_jmp = DISAS_WFE;
 return;
 case 4: /* SEV */
 case 5: /* SEVL */
diff --git a/target/arm/translate.c b/target/arm/translate.c
index f120932..130ab50 100644
--- a/target/arm/translate.c
+++ b/target/arm/translate.c
@@ -4531,14 +4531,6 @@ static void gen_exception_return(DisasContext *s, 
TCGv_i32 pc)
 gen_rfe(s, pc, load_cpu_field(spsr));
 }

-/*
- * For WFI we will halt the vCPU until an IRQ. For WFE and YIELD we
- * only call the helper when running single threaded TCG code to ensure
- * the next round-robin scheduled vCPU gets a crack. In MTTCG mode we
- * just skip this instruction. Currently the SEV/SEVL instructions
- * which are *one* of many ways to wake the CPU from WFE are not
- * implemented so we can't sleep like WFI does.
- */
 static void gen_nop_hint(DisasContext *s, int val)
 {
 switch (val) {
@@ -4548,20 +4540,16 @@ static void gen_nop_hint(DisasContext *s, int val)
  * spin unnecessarily we would need to do something more involved.
  */
 case 1: /* yield */
-if (!(tb_cflags(s->base.tb) & CF_PARALLEL)) {
-gen_set_pc_im(s, s->pc);
-s->base.is_jmp = DISAS_YIELD;
-}
+gen_set_pc_im(s, s->pc);
+s->base.is_jmp = DISAS_YIELD;
 break;
 case 3: /* wfi */
 gen_set_pc_im(s, s->pc);
 s->base.is_jmp = DISAS_WFI;
 break;
 case 2: /* wfe */
-    if (!(tb_cflags(s->base.tb) & CF_PARALLEL)) {
-gen_s

Re: [OE-core] Need arm64/qemu help

[Victor Kamensky]

On Sun, 11 Mar 2018, Peter Maydell wrote:


On 11 March 2018 at 00:11, Victor Kamensky <kamen...@cisco.com> wrote:

Hi Richard, Ian,

Any progress on the issue? In case if not, I am adding few Linaro guys
who work on aarch64 qemu. Maybe they can give some insight.


No immediate answers, but we might be able to have a look
if you can provide a repro case (image, commandline, etc)
that doesn't require us to know anything about OE and your
build/test infra to look at.


Peter, thank you! Appreciate your attention and response to
this. It is fair ask, I should have tried to narrow test
case down before punting it to you guys.


(QEMU's currently just about
to head into codefreeze for our next release, so I'm a bit
busy for the next week or so. Alex, do you have time to
take a look at this?)

Does this repro with the current head-of-git QEMU?


I've tried head-of-git QEMU (Mar 9) on my ubuntu-16.04
with the same target Image and rootfs I could not reproduce
the issue.

I've started to play around more trying to reduce the test
case. In my setup with OE qith qemu 2.11.1, if I just passed
'-serial sdtio' or '-nographic', instead of '-serial mon:vc'
- with all things the same image boots fine.

So, I started to suspect, even if problem manifests itself
as some functional failure of qemu, the issue could be some
nasty memory corruption of some qemu operational data.
And since qemu pull bunch of dependent
libraries, problem might be not even in qemu.

I realized that in OE in order to disconnect itself from
underlying host, OE builds a lot of its own "native"
libaries and OE qemu uses them. So I've tried to build
head-of-git QEMU but with all native libraries that OE
builds - now such combinations hangs in the same way.

Also I noticed that OE qemu is built with SDL (v1.2),
and libsdl is one that reponsible for '-serial mon:vc'
handling. And I noticed in default OE conf/local.conf
the following statements:

#
# Qemu configuration
#
# By default qemu will build with a builtin VNC server where graphical 
output can be
# seen. The two lines below enable the SDL backend too. By default 
libsdl-native will
# be built, if you want to use your host's libSDL instead of the minimal 
libsdl built

# by libsdl-native then uncomment the ASSUME_PROVIDED line below.
PACKAGECONFIG_append_pn-qemu-native = " sdl"
PACKAGECONFIG_append_pn-nativesdk-qemu = " sdl"
#ASSUME_PROVIDED += "libsdl-native"

I've tried to build against my host's libSDL and uncommented
above line. It actually failed to build, because my host libSDL
were not happy about ncurses native libraries, so I ended up
adding this as well:

ASSUME_PROVIDED += "ncurses-native"

After that I had to rebuild qemu-native and qemu-helper-native.
With resulting qemu and the same target files, image boots
OK.

With such nasty corruption problem, it always hard to say for
sure, it maybe just timing changes .. , but now it seems it
somewhat points to some issue in OE libsdl version ... And
still it is fairly bizarre, libsdl
that in OE (1.2.15) is the same that I have on my ubuntu
machine and there is no additional patches for it in OE,
although configure options might be quite different.

Thanks,
Victor


If for experiment sake I disable loop that tries to find
jiffies transition. I.e have something like this:

diff --git a/lib/raid6/algos.c b/lib/raid6/algos.c
index 4769947..e0199fc 100644
--- a/lib/raid6/algos.c
+++ b/lib/raid6/algos.c
@@ -166,8 +166,12 @@ static inline const struct raid6_calls
*raid6_choose_gen(

preempt_disable();
j0 = jiffies;
+#if 0
while ((j1 = jiffies) == j0)
cpu_relax();
+#else
+j1 = jiffies;
+#endif /* 0 */
while (time_before(jiffies,
j1 +
(1<<RAID6_TIME_JIFFIES_LG2))) {
(*algo)->gen_syndrome(disks, PAGE_SIZE,
*dptrs);
@@ -189,8 +193,12 @@ static inline const struct raid6_calls
*raid6_choose_gen(

preempt_disable();
j0 = jiffies;
+#if 0
while ((j1 = jiffies) == j0)
cpu_relax();
+#else
+j1 = jiffies;
+#endif /* 0 */
while (time_before(jiffies,
j1 +
(1<<RAID6_TIME_JIFFIES_LG2))) {
(*algo)->xor_syndrome(disks, start, stop,

Image boots fine after that.

I.e it looks as some strange effect in aarch64 qemu that seems does not
progress jiffies and code stuck.



Another observation is that if I put breakpoint for example
in do_timer, it actually hits the breakpoint, ie timer interrupt
happens in this case, and strangely raid6_choose_gen sequence
does progress, ie debugger breakpoints make this case unstuck.
Actually several pressing Ctrl

Re: [OE-core] Need arm64/qemu help

[Victor Kamensky]

Hi Richard, Ian,

Any progress on the issue? In case if not, I am adding few Linaro guys
who work on aarch64 qemu. Maybe they can give some insight.

I was able to reproduce on my system and I
and look at it under gdb. It seems that some strange aarch64
percularity might be in play. Details inline, root cause is still
not clear.

On Sat, 3 Mar 2018, Ian Arkver wrote:


On 03/03/18 10:51, Ian Arkver wrote:

On 03/03/18 09:00, Richard Purdie wrote:

Hi,

I need some help with a problem we keep seeing:

https://autobuilder.yocto.io/builders/nightly-arm64/builds/798

Basically, now and again, for reasons we don't understand, all the
sanity tests fail for qemuarm64.

I've poked at this a bit and if I go in onto the failed machine and run
this again, they work, using the same image, kernel and qemu binaries.
We've seen this on two different autobuilder infrastructure on varying
host OSs. They always seem to fail all three at once.

Whilst this was a mut build, I saw this repeat three builds in a row on
the new autobuilder we're setting up with master.

The kernels always seem to hang somewhere around the:

| [0.766079] raid6: int64x1  xor()   302 MB/s
| [0.844597] raid6: int64x2  gen()   675 MB/s


I believe this is related to btrfs and comes from having btrfs compiled 
in to the kernel. You could maybe side-step the problem (and hence leave 
it lurking) by changing btrfs to a module.


Actually, this comes from a library (lib/raid6), and in 4.14.y's arm64 
defconfig BTRFS is already a module, so please disregard my hack suggestion.


Indeed, in my case when I run qemu with enabled remote gdbserver, and
in kernel hang boot case I press Ctrl-C and drop into gdb I see the
following traceback:

(gdb) bt
#0  vectors ()
at 
/wd6/oe/20180304/systemtap-oe-sysroot/build/tmp-glibc/work-shared/qemuarm64/kernel-source/arch/arm64/kernel/entry.S:376
#1  0xff80089a2ff4 in raid6_choose_gen (disks=, 
dptrs=)
at 
/wd6/oe/20180304/systemtap-oe-sysroot/build/tmp-glibc/work-shared/qemuarm64/kernel-source/lib/raid6/algos.c:190
#2  raid6_select_algo ()
at 
/wd6/oe/20180304/systemtap-oe-sysroot/build/tmp-glibc/work-shared/qemuarm64/kernel-source/lib/raid6/algos.c:253
#3  0xff8008083b8c in do_one_initcall (fn=0xff80089a2e64 
)
at 
/wd6/oe/20180304/systemtap-oe-sysroot/build/tmp-glibc/work-shared/qemuarm64/kernel-source/init/main.c:832
#4  0xff8008970e80 in do_initcall_level (level=)
at 
/wd6/oe/20180304/systemtap-oe-sysroot/build/tmp-glibc/work-shared/qemuarm64/kernel-source/init/main.c:898
#5  do_initcalls () at 
/wd6/oe/20180304/systemtap-oe-sysroot/build/tmp-glibc/work-shared/qemuarm64/kernel-source/init/main.c:906
#6  do_basic_setup () at 
/wd6/oe/20180304/systemtap-oe-sysroot/build/tmp-glibc/work-shared/qemuarm64/kernel-source/init/main.c:924
#7  kernel_init_freeable ()
at 
/wd6/oe/20180304/systemtap-oe-sysroot/build/tmp-glibc/work-shared/qemuarm64/kernel-source/init/main.c:1073
#8  0xff80087a2e00 in kernel_init (unused=)
at 
/wd6/oe/20180304/systemtap-oe-sysroot/build/tmp-glibc/work-shared/qemuarm64/kernel-source/init/main.c:999
#9  0xff80080850ec in ret_from_fork ()
at 
/wd6/oe/20180304/systemtap-oe-sysroot/build/tmp-glibc/work-shared/qemuarm64/kernel-source/arch/arm64/kernel/entry.S:994
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
(gdb) x /10i $pc - 12
   0xff8008082274 :  nop
   0xff8008082278 :  nop
   0xff800808227c :  nop
=> 0xff8008082280 :   sub sp, sp, #0x140
   0xff8008082284 :  add sp, sp, x0
   0xff8008082288 :  sub x0, sp, x0
   0xff800808228c :  tbnzw0, #14, 0xff800808229c 

   0xff8008082290 :  sub x0, sp, x0
   0xff8008082294 :  sub sp, sp, x0
   0xff8008082298 :  b   0xff8008082fc0 
(gdb) f 1
#1  0xff80089a2ff4 in raid6_choose_gen (disks=, 
dptrs=)
at 
/wd6/oe/20180304/systemtap-oe-sysroot/build/tmp-glibc/work-shared/qemuarm64/kernel-source/lib/raid6/algos.c:190
190 preempt_disable();
(gdb) x /12i $pc - 12
   0xff80089a2fe8 :cbz x0, 0xff80089a3098 

   0xff80089a2fec :mov w0, #0x1  
  // #1
   0xff80089a2ff0 :bl  0xff80080cc498 

=> 0xff80089a2ff4 : ldr x0, [x23, #2688]
   0xff80089a2ff8 :ldr x5, [x23, #2688]
   0xff80089a2ffc :cmp x0, x5
   0xff80089a3000 :b.ne0xff80089a300c 
  // b.any
   0xff80089a3004 :yield
   0xff80089a3008 :b   0xff80089a2ff8 

Re: [OE-core] [RFC PATCH 0/5] systemtap adding sysroot, cross compiling of user land related scripts

[Victor Kamensky]

Hi Alex,

On Tue, 6 Mar 2018, Alexander Kanavin wrote:


On 03/06/2018 07:50 PM, Victor Kamensky wrote:

I am a fan of SystemTap and big fan of OE. Systemtap and OE use case I have
on my mind is the following:


Hello Victor,

even though I'm listed as the maintainer of systemtap, I do not actually use 
it or know anything about it - my responsibility ends with keeping it up to 
date and fixing build errors.


Fair enough. It is understood.

So I can't comment on the essence of these 
patches. I do suggest however that you first get as many patches as possible 
into the upstream, as carrying them in oe-core adds a significant maintenance 
burden.


Yes, it is understood. I am actively working, as we speak,
with SystemTap folks on putting sysroot related patches into SystemTap.

Also, if it's possible to merge systemtap-utils into the main systemtap 
recipe (as a binary package),


I've set to write response why systemtap-utils is really
needed and ended up doing below :). Does it look acceptable
direction to you?


From 808fbe3c4b64c734258da08e92fb367a8a20939b Mon Sep 17 00:00:00 2001

From: Victor Kamensky <kamen...@cisco.com>
Date: Wed, 7 Mar 2018 18:33:33 -0800
Subject: [PATCH 08/11] systemtap: create translator packageconfig

For cases when systemap module compilation happens on host in
cross-compilation mode, and it is desirable to minimize systemtap
presense on target we need to have just smallest possible set of
utilties that are required to run compiled modules.

Introduce new "translator" PACKAGECONFIG, if it is not set
it would mean that just minimal set of run-time utilities will
be included in the package.

For run-time only systemtap build variant use
PACKAGECONFIG_pn-systemtap = "" or
PACKAGECONFIG_pn-systemtap = "monitor"

Suggested-by: Taras Kondratiuk <takon...@cisco.com>
Signed-off-by: Victor Kamensky <kamen...@cisco.com>
---
 meta/recipes-kernel/systemtap/systemtap_git.bb | 15 +++
 1 file changed, 11 insertions(+), 4 deletions(-)

diff --git a/meta/recipes-kernel/systemtap/systemtap_git.bb 
b/meta/recipes-kernel/systemtap/systemtap_git.bb
index 475b207..b280f58 100644
--- a/meta/recipes-kernel/systemtap/systemtap_git.bb
+++ b/meta/recipes-kernel/systemtap/systemtap_git.bb
@@ -3,9 +3,7 @@ HOMEPAGE = "https://sourceware.org/systemtap/;

 require systemtap_git.inc

-DEPENDS = "boost elfutils"
-
-RDEPENDS_${PN} += "python3-core bash perl"
+DEPENDS = "elfutils"

 EXTRA_OECONF += "--with-libelf=${STAGING_DIR_TARGET} --without-rpm \
 --without-nss --without-avahi --without-dyninst \
@@ -18,7 +16,8 @@ STAP_DOCS ?= "--disable-docs --disable-publican 
--disable-refdocs"

 EXTRA_OECONF += "${STAP_DOCS} "

-PACKAGECONFIG ??= "sqlite monitor python3-probes"
+PACKAGECONFIG ??= "translator sqlite monitor python3-probes"
+PACKAGECONFIG[translator] = 
"--enable-translator,--disable-translator,boost,python3-core bash perl"
 PACKAGECONFIG[libvirt] = "--enable-libvirt,--disable-libvirt,libvirt"
 PACKAGECONFIG[sqlite] = "--enable-sqlite,--disable-sqlite,sqlite3"
 PACKAGECONFIG[monitor] = "--enable-monitor,--disable-monitor,ncurses json-c"
@@ -26,4 +25,12 @@ PACKAGECONFIG[python3-probes] = 
"--with-python3-probes,--without-python3-probes,

 inherit autotools gettext pkgconfig distutils3-base

+do_install_append () {
+   if [ ! -f ${D}${bindir}/stap ]; then
+  # translator disabled case, need to leave only minimal runtime
+  rm -rf ${D}${datadir}/${PN}
+  rm ${D}${libexecdir}/${PN}/stap-env
+   fi
+}
+
 BBCLASSEXTEND = "nativesdk"
--
2.7.4


and merge together crosstap and crosstap2,  please do that.


I added code into crosstap2 to provide backward compatible
crosstap legecy invocation interface. I plan to post new crosstap2 code
as total backward compatible replacement of crosstap with superseding 
functionality.


The only controvercial item left from my original ask is
IMAGE_GEN_COMBINED_DEBUGFS build option that is supposed to
build rootfs-dbg containing combined target and debug symbols.

Thanks,
Victor


Alex


--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [RFC PATCH 5/5] crosstap2: wrapper to build systemtap script on host against given target

[Victor Kamensky]

crosstap2 is similar to existing crosstap shell script, but unlike crosstap
script that can handle only kernel SystemTap scripts, crosstap2 can also
deal with user land related SystemTap scripts. It discovers and set up
proper --sysroot arguments for stap invocation.

The logic of handling of user-land related cross build arguments is a bit
more complicated. It seemed that to implement it in existing crosstap
script would be a stretch, so crosstap2 is written in python.

Signed-off-by: Victor Kamensky <kamen...@cisco.com>
---
 scripts/crosstap2 | 422 ++
 1 file changed, 422 insertions(+)
 create mode 100755 scripts/crosstap2

diff --git a/scripts/crosstap2 b/scripts/crosstap2
new file mode 100755
index 000..c5209d2
--- /dev/null
+++ b/scripts/crosstap2
@@ -0,0 +1,422 @@
+#!/usr/bin/env python3
+#
+# Build a systemtap script for a given image.
+#
+# Effectively script extracts needed information from set of
+# 'bitbake -e' commands and contructs proper invocation of stap on
+# host to build systemtap script for a given target.
+#
+# By default script will compile scriptname.ko that could be copied
+# to taget and activated with 'staprun scriptname.ko' command. Or if
+# --remote user@hostname option is specified script will build, load
+# execute script on target.
+#
+# This script is very similar and inspired by crosstap shell script.
+# The major difference that this script supports user-land related
+# systemtap script, whereas crosstap could deal only with scripts
+# related to kernel.
+#
+# Copyright (c) 2018, Cisco Systems.
+# All rights reserved.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
+# See the GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+
+import sys
+import re
+import subprocess
+import os
+
+from optparse import OptionParser
+
+class Stap(object):
+def __init__(self, script, module, remote):
+self.script = script
+self.module = module
+self.remote = remote
+self.stap = None
+self.sysroot = None
+self.runtime = None
+self.tapset = None
+self.arch = None
+self.cross_compile = None
+self.kernel_release = None
+self.target_path = None
+self.target_ld_library_path = None
+
+if not self.remote:
+if not self.module:
+# derive module name from script
+self.module = os.path.basename(self.script)
+if self.module[-4:] == ".stp":
+self.module = self.module[:-4]
+# replace - if any with _
+self.module = self.module.replace("-", "_")
+
+def command(self, args):
+ret = []
+ret.append(self.stap)
+
+if self.remote:
+ret.append("--remote")
+ret.append(self.remote)
+else:
+ret.append("-p4")
+ret.append("-m")
+ret.append(self.module)
+
+ret.append("--sysroot")
+ret.append(self.sysroot)
+
+ret.append("-a")
+ret.append(self.arch)
+
+ret.append("-B")
+ret.append("CROSS_COMPILE=" + self.cross_compile)
+
+ret.append("-r")
+ret.append(self.kernel_release)
+
+ret.append("-I")
+ret.append(self.tapset)
+
+ret.append("-R")
+ret.append(self.runtime)
+
+ret.append("--sysenv=PATH=" + self.target_path)
+ret.append("--sysenv=LD_LIBRARY_PATH=" + self.target_ld_library_path)
+
+ret = ret + args
+
+ret.append(self.script)
+return ret
+
+def additional_environment(self):
+ret = {}
+ret["SYSTEMTAP_DEBUGINFO_PATH"] = "+:.debug:build"
+return ret
+
+def environment(self):
+ret = os.environ.copy()
+additional = self.additional_environment()
+for e in additional:
+ret[e] = additional[e]
+return ret
+
+def display_command(self, args):
+additional_env = self.additional_environment()
+command = self.command(args)
+
+print("#!/bin/sh")
+for e in additional_env:
+print("export %s=\"%s\"" % (e, additional_env[e]))
+print(" ".join(command))
+
+class B

[OE-core] [RFC PATCH 4/5] systemtap: introduce utils variant of systemtap package build

[Victor Kamensky]

In case of small target images it is too expensive to include
full stap package into the image. Because default systemtap
package include SystemTap translater and have very deep dependencies
on variety of libraries. And in cross compiling environment
compilation of SystemTap kernel modules can happen on host for
a given target using --sysroot and other proper options.

In order to deploy resulting SystemTap .ko module, one need small
set of SystemTap runtime tools like staprun and stapio.

systemtap-utils introduces special "small" variant of systemtap
build where only SystemTap run-time utilties are built. As result
addition on target is quite small.

Signed-off-by: Victor Kamensky <kamen...@cisco.com>
---
 .../systemtap/systemtap-utils_git.bb   | 31 ++
 1 file changed, 31 insertions(+)
 create mode 100644 meta/recipes-kernel/systemtap/systemtap-utils_git.bb

diff --git a/meta/recipes-kernel/systemtap/systemtap-utils_git.bb 
b/meta/recipes-kernel/systemtap/systemtap-utils_git.bb
new file mode 100644
index 000..3359257
--- /dev/null
+++ b/meta/recipes-kernel/systemtap/systemtap-utils_git.bb
@@ -0,0 +1,31 @@
+SUMMARY = "Small set only runtime utilities for SystemTap"
+HOMEPAGE = "https://sourceware.org/systemtap/;
+
+require systemtap_git.inc
+
+FILESEXTRAPATHS =. "${FILE_DIRNAME}/systemtap:"
+
+DEPENDS = "elfutils"
+
+# should not be used along with complete systemtap
+RCONFLICTS_${PN} = "systemtap"
+
+EXTRA_OECONF += "--with-libelf=${STAGING_DIR_TARGET} --without-rpm \
+--without-nss --without-avahi --without-dyninst \
+--disable-server --disable-grapher --enable-prologues \
+--with-python3 --without-python2-probes \
+--disable-sdt-probes --disable-translator \
+--disable-libvirt --disable-sqlite --disable-monitor \
+--without-python3-probes \
+--disable-docs --disable-publican --disable-refdocs \
+ac_cv_prog_have_javac=no \
+ac_cv_prog_have_jar=no "
+
+do_install_append () {
+rm -rf ${D}${datadir}/systemtap
+rm -rf ${D}${includedir}
+# remove bash dependency we don't need it anyway
+rm ${D}${libexecdir}/systemtap/stap-env
+}
+
+inherit autotools gettext
-- 
2.7.4

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [RFC PATCH 3/5] image: add IMAGE_GEN_COMBINED_DEBUGFS build option

[Victor Kamensky]

There is IMAGE_GEN_DEBUGFS="1" variable that enables build of
additional rootfs-dbg and additional archive that contains
complimentary symbols files for a given image. But the issue
with this resulting directory and tarball that before use it
has to be combined with original image content. It is required
since all cross debugging tools like gdb, perf, and systemtap
need file system that contains both target executables/libraries
and their symbols. Those tools need to find executable/library
first and through it debuglink note find corresponding symbols
file.

IMAGE_GEN_DEBUGFS="1" variable as storage strategy may have
some sense it removes duplicate content, but from engineer
work flow point of view it requires extra steps of recombining,
unpacking two tarball archives. I.e there is no directory in
image build workspace that can be used to run gdb, perf, and
systemtap.

In addition to IMAGE_GEN_DEBUGFS variable new variant of
the same functionality IMAGE_GEN_COMBINED_DEBUGFS is introduced.
Unlike IMAGE_GEN_DEBUGFS additional rootfs-dbg and additional
tarball would contain both all target material plus symbols.
Resulting rootfs-dbg directory in build workspace can be
directly used by all cross debugging tools like gdb, perf, and
systemtap.

Signed-off-by: Victor Kamensky <kamen...@cisco.com>
---
 meta/classes/image-prelink.bbclass | 13 +
 meta/classes/image.bbclass | 11 +--
 meta/lib/oe/rootfs.py  | 59 +-
 3 files changed, 55 insertions(+), 28 deletions(-)

diff --git a/meta/classes/image-prelink.bbclass 
b/meta/classes/image-prelink.bbclass
index f3bb68b..5470e63 100644
--- a/meta/classes/image-prelink.bbclass
+++ b/meta/classes/image-prelink.bbclass
@@ -35,6 +35,11 @@ prelink_image () {
 
dynamic_loader=$(linuxloader)
 
+if [ "$IMAGE_GEN_COMBINED_DEBUGFS" = "1" ]; then
+cp ${IMAGE_ROOTFS}${sysconfdir}/prelink.conf 
${IMAGE_ROOTFS}-dbg${sysconfdir}/prelink.conf
+cp ${IMAGE_ROOTFS}${sysconfdir}/ld.so.conf 
${IMAGE_ROOTFS}-dbg${sysconfdir}/ld.so.conf
+fi
+
# prelink!
if [ "$BUILD_REPRODUCIBLE_BINARIES" = "1" ]; then
bbnote " prelink: BUILD_REPRODUCIBLE_BINARIES..."
@@ -44,8 +49,16 @@ prelink_image () {
export PRELINK_TIMESTAMP=$REPRODUCIBLE_TIMESTAMP_ROOTFS
fi
${STAGING_SBINDIR_NATIVE}/prelink --root ${IMAGE_ROOTFS} -am -N 
-c ${sysconfdir}/prelink.conf --dynamic-linker $dynamic_loader
+if [ "$IMAGE_GEN_COMBINED_DEBUGFS" = "1" ]; then
+ bbnote " prelink: IMAGE_GEN_COMBINED_DEBUGFS..."
+${STAGING_SBINDIR_NATIVE}/prelink --root 
${IMAGE_ROOTFS}-dbg -am -N -c ${sysconfdir}/prelink.conf --dynamic-linker 
$dynamic_loader
+fi
else
${STAGING_SBINDIR_NATIVE}/prelink --root ${IMAGE_ROOTFS} -amR 
-N -c ${sysconfdir}/prelink.conf --dynamic-linker $dynamic_loader
+if [ "$IMAGE_GEN_COMBINED_DEBUGFS" = "1" ]; then
+ bbnote " prelink: IMAGE_GEN_COMBINED_DEBUGFS..."
+${STAGING_SBINDIR_NATIVE}/prelink --root 
${IMAGE_ROOTFS}-dbg -amR -N -c ${sysconfdir}/prelink.conf --dynamic-linker 
$dynamic_loader
+fi
fi
 
# Remove the prelink.conf if we had to add it.
diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index 23ed53d..3d1b9e4 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -28,6 +28,12 @@ IMAGE_FEATURES[validitems] += "debug-tweaks read-only-rootfs 
empty-root-password
 # Generate companion debugfs?
 IMAGE_GEN_DEBUGFS ?= "0"
 
+# Generate companion combined debugfs? The difference from IMAGE_GEN_DEBUGFS
+# is that in case of IMAGE_GEN_COMBINED_DEBUGFS resulting debufs contains full
+# copy of original symbols plus -dbg packages, whereas in case of 
IMAGE_GEN_DEBUGFS
+# it contains only -dbg packages
+IMAGE_GEN_COMBINED_DEBUGFS ?= "0"
+
 # rootfs bootstrap install
 ROOTFS_BOOTSTRAP_INSTALL = "run-postinsts"
 
@@ -117,7 +123,7 @@ def rootfs_variables(d):
  
'IMAGE_ROOTFS_MAXSIZE','IMAGE_NAME','IMAGE_LINK_NAME','IMAGE_MANIFEST','DEPLOY_DIR_IMAGE','IMAGE_FSTYPES','IMAGE_INSTALL_COMPLEMENTARY','IMAGE_LINGUAS',
  
'MULTILIBRE_ALLOW_REP','MULTILIB_TEMP_ROOTFS','MULTILIB_VARIANTS','MULTILIBS','ALL_MULTILIB_PACKAGE_ARCHS','MULTILIB_GLOBAL_VARIANTS','BAD_RECOMMENDATIONS','NO_RECOMMENDATIONS',
  
'PACKAGE_ARCHS','PACKAGE_CLASSES','TARGET_VENDOR','TARGET_ARCH','TARGET_OS','OVERRIDES','BBEXTENDVARIANT','FEED_DEPLOYDIR_BASE_URI','INTERCEPT_DIR','USE_DEVFS',
- 'CONVERSIONTYPES', 'IMAGE_GEN_DEBUGFS', 'ROOTFS_RO_UNNEEDED', 
'IMGDEPLOYDIR', 'PACKAGE_EXCLUDE_COM

[OE-core] [RFC PATCH 0/5] systemtap adding sysroot, cross compiling of user land related scripts

[Victor Kamensky]

Hi Guys,

I am a fan of SystemTap and big fan of OE. Systemtap and OE use case I have
on my mind is the following:

1) small embedded target image built, because of resource constraint such
target could not run full blown systemtap on, it would not have resources
to have debug symbols, toolchain, kernel sources, etc installed on target,
memory size available to to SystemTap script build, etc

2) on host target image is build with debug symbols available somewhere
on the host

3) systemtap operates in cross compilation mode, builds targetscript.ko module,
which could be copied to target device

4) on target tiny systemtap runtime, like staprun, stapio, utilities are
present and they are used to activate systemtap targetscript.ko and observe
SystemTap in action.

In fact OE already has scripts/crosstap script that provides similar to above
experience. But the scripts one can have with crosstap can deal only with
kernel code. If one would like to write SystemTap script that involves
user land process probes crosstap script will not work.

In part the problem was that in SystemTap itself --sysroot options was
effectively broken or never worked. So it was not possible to build user-land
related script in cross compiled mode. I worked on SystemTap fixing --sysroot 
option support and I posted these fixes on systemtap mailing list [1].

Now with --sysroot option support fixed in SystemTap it comes to the
question how it could be integrated into OE related engineers work flow. And
here I need team's feedback. These patch series implements proposal for such
improved SystemTap OE integration, but some parts can cause controversy.

Here is patches overview, with indication of my concerns and ask for
feedback:

  Revert "systemtap: Cross compilation fix"

  Remove previous OE specific patch. It superseded by --sysroot support
  patch series


  systemtap: support --sysroot option in variety of situations in cross
build

  effectively [1] integrated with OE, these patches will be discussed
  with SystemTap guys directly.


  image: add IMAGE_GEN_COMBINED_DEBUGFS build option

  this one may cause some controversy. For SystemTap to build user-land
  related script, it needs target sysroot tree overlayed with debug
  symbols. OE already has IMAGE_GEN_DEBUGFS option implemented but
  it produces split target and debug symbols tarballs that have to be
  recombined. I claim step of recombination is a drug on developer
  work flow. IMAGE_GEN_COMBINED_DEBUGFS is similar to IMAGE_GEN_DEBUGFS,
  but resulting rootfs-dbg would contain both target and symbols, so
  it could be used right away. Also I feel that implementation of
  IMAGE_GEN_COMBINED_DEBUGFS option might not be very elegant, especially
  with prelink related issue that changes target binaries


  systemtap: introduce utils variant of systemtap package build

  systemtap-utils recipe is special systemtap build that would compile
  only utilities and have minimal dependencies possible. I considered to
  split out those utilities on packaging step of systemtap but with
  dependencies description, it got very messy, so I decided to go
  with separate conflicting with systemtap recipe


  crosstap2: wrapper to build systemtap script on host against given
target

  Wrapper script similar to crosstap that take SystemTap script and
  image name, retrieve rest of required information from bitbake -e
  command and construct stap invocation with all proper arguments
  passed needed for cross compilation. Script is implemented in python
  it has more advanced logic then existing crosstap tool. But one may
  ask why I did not improved existing crosstap script - I felt that
  new requirements is too much to be implemented in shell script and
  in backward compatible mode without breaking historic interface to
  crosstap.


If someone interested in full patch series my OE tree with these fixes
is here [2]. It was tested on variety of qemu related machines. Steps that
shows how whole work flow OE and SystemTap works are here [3]. Examples of
crosstap2 invocations are here [4].

[1] https://sourceware.org/ml/systemtap/2018-q1/msg00069.html

[2] https://github.com/victorkamensky/openembedded-core

[3] https://github.com/victorkamensky/systemtap-oe-sysroot-manifest

[4] 
https://github.com/victorkamensky/systemtap-oe-sysroot-manifest/blob/master/EXAMPLES

Victor Kamensky (5):
  Revert "systemtap: Cross compilation fix"
  systemtap: support --sysroot option in variety of situations in cross
build
  image: add IMAGE_GEN_COMBINED_DEBUGFS build option
  systemtap: introduce utils variant of systemtap package build
  crosstap2: wrapper to build systemtap script on host against given
target

 meta/classes/image-prelink.bbclass |  13 +
 meta/classes/image.bbclass |  11 +-
 m

[OE-core] [RFC PATCH 1/5] Revert "systemtap: Cross compilation fix"

[Victor Kamensky]

This reverts commit 787bed708676fc04aee2850825e803273152f657.

Signed-off-by: Victor Kamensky <kamen...@cisco.com>
---
 .../systemtap/systemtap/system_map_location.patch  | 23 --
 meta/recipes-kernel/systemtap/systemtap_git.inc|  1 -
 2 files changed, 24 deletions(-)
 delete mode 100644 
meta/recipes-kernel/systemtap/systemtap/system_map_location.patch

diff --git a/meta/recipes-kernel/systemtap/systemtap/system_map_location.patch 
b/meta/recipes-kernel/systemtap/systemtap/system_map_location.patch
deleted file mode 100644
index 013af5c..000
--- a/meta/recipes-kernel/systemtap/systemtap/system_map_location.patch
+++ /dev/null
@@ -1,23 +0,0 @@
-systemtap: Cross compilation fix
-
-This is a cross compilation fix. It allows systemtap to find
-the kernel map file in the right place, i.e. in the kernel build tree.
-Without this fix it takes a map file from the build host, if available.
-
-Upstream-Status: Pending
-
-Signed-off-by: Mikhail Durnev <mikhail_dur...@mentor.com>
-
-Index: git/session.cxx
-===
 git.orig/session.cxx
-+++ git/session.cxx
-@@ -1634,7 +1634,7 @@ systemtap_session::parse_kernel_function
-   clog << _F("Kernel symbol table %s unavailable, (%s)",
-  system_map_path.c_str(), strerror(errno)) << endl;
- 
--  system_map_path = "/boot/System.map-" + kernel_release;
-+  system_map_path = kernel_build_tree + "/System.map-" + kernel_release;
-   system_map.clear();
-   system_map.open(system_map_path.c_str(), ifstream::in);
-   if (! system_map.is_open())
diff --git a/meta/recipes-kernel/systemtap/systemtap_git.inc 
b/meta/recipes-kernel/systemtap/systemtap_git.inc
index f51bd28..af5b004 100644
--- a/meta/recipes-kernel/systemtap/systemtap_git.inc
+++ b/meta/recipes-kernel/systemtap/systemtap_git.inc
@@ -4,7 +4,6 @@ SRCREV = "4051c70c9318c837981384cbb23f3e9eb1bd0892"
 PV = "3.2"
 
 SRC_URI = "git://sourceware.org/git/systemtap.git \
-   file://system_map_location.patch \
file://configure-allow-to-disable-libvirt.patch \
file://x32_abi_time.patch \
file://monitor-option.patch \
-- 
2.7.4

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [RFC PATCH 2/5] systemtap: support --sysroot option in variety of situations in cross build

[Victor Kamensky]

For details on issues fixed please look at commit message of individual
patches.

Upstream-Status: Submitted [system...@sourceware.org]

Change-Id: Id2fd0fbddb33e4a4387b732005a6e902a5c6129c
Signed-off-by: Victor Kamensky <kamen...@cisco.com>
---
 ...-short-release-r-option-handling-in-case-.patch |  61 
 ...-short-release-r-option-handling-in-case-.patch |  68 +
 ...uginfo-lookup-with-sysroot-case-do-not-re.patch |  97 +
 ...ay-adding-sysroot-path-to-module-name-in-.patch | 124 
 ...dle-symbolic-links-with-absolute-name-rel.patch | 159 +
 ...p_umodule_relocate-needs-receive-target-f.patch |  58 
 meta/recipes-kernel/systemtap/systemtap_git.inc|   6 +
 7 files changed, 573 insertions(+)
 create mode 100644 
meta/recipes-kernel/systemtap/systemtap/0001-sysroot-fix-short-release-r-option-handling-in-case-.patch
 create mode 100644 
meta/recipes-kernel/systemtap/systemtap/0002-sysroot-fix-short-release-r-option-handling-in-case-.patch
 create mode 100644 
meta/recipes-kernel/systemtap/systemtap/0003-sysroot-debuginfo-lookup-with-sysroot-case-do-not-re.patch
 create mode 100644 
meta/recipes-kernel/systemtap/systemtap/0004-sysroot-delay-adding-sysroot-path-to-module-name-in-.patch
 create mode 100644 
meta/recipes-kernel/systemtap/systemtap/0005-sysroot-handle-symbolic-links-with-absolute-name-rel.patch
 create mode 100644 
meta/recipes-kernel/systemtap/systemtap/0006-sysroot-_stp_umodule_relocate-needs-receive-target-f.patch

diff --git 
a/meta/recipes-kernel/systemtap/systemtap/0001-sysroot-fix-short-release-r-option-handling-in-case-.patch
 
b/meta/recipes-kernel/systemtap/systemtap/0001-sysroot-fix-short-release-r-option-handling-in-case-.patch
new file mode 100644
index 000..607c2cd
--- /dev/null
+++ 
b/meta/recipes-kernel/systemtap/systemtap/0001-sysroot-fix-short-release-r-option-handling-in-case-.patch
@@ -0,0 +1,61 @@
+From 8fccac2483354a8a3e2419c36c867c81045dbe7f Mon Sep 17 00:00:00 2001
+From: Victor Kamensky <kamen...@cisco.com>
+Date: Thu, 8 Feb 2018 20:00:46 -0800
+Subject: [PATCH 1/6] sysroot: fix short release -r option handling in case of
+ sysroot read System.map symbols
+
+Add sysroot to system_map_path + "/boot/System.map" case. Otherwise
+stap tries to look symbols on host system and it produce error like this:
+
+> Kernel symbol table /boot/System.map-4.9.78-yocto-standard unavailable, (No 
such file or directory)
+
+Here are steps to reproduce the issue. Note  is produced by yocto
+poky build. Under /lib/modules/4.9.78-yocto-standard/build there
+is no System.map file; instead it is installed under /boot directory.
+
+[kamensky@coreos-lnx2 tests]$ cat meminfo.stp
+probe kernel.function("meminfo_proc_show") {
+println("meminfo_proc_show called")
+}
+[kamensky@coreos-lnx2 tests]$ export SYSTEMTAP_DEBUGINFO_PATH=+:.debug:build
+[kamensky@coreos-lnx2 tests]$ /home/wd8/systemtap/20180208_2/packages/bin/stap 
--sysroot=/home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs
 -a x86_64 -r 4.9.78-yocto-standard -B CROSS_COMPILE=x86_64-poky-linux- 
--sysenv=PATH=/usr/bin:/bin:/usr/sbin:/sbin 
--sysenv=LD_LIBRARY_PATH=/lib:/usr/lib -p4 -v -v -m meminfo meminfo.stp
+Systemtap translator/driver (version 3.3/0.170, commit 
release-3.2-92-g26c5e3c03c94 + changes)
+Copyright (C) 2005-2017 Red Hat, Inc. and others
+This is free software; see the source for copying conditions.
+tested kernel versions: 2.6.18 ... 4.14-rc4
+enabled features: BPF PYTHON2 PYTHON3 LIBXML2 NLS
+Created temporary directory "/tmp/staprq4IGE"
+Session arch: x86_64 release: 4.9.78-yocto-standard
+Kernel symbol table 
/home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs//lib/modules/4.9.78-yocto-standard/build/System.map
 unavailable, (No such file or directory)
+Kernel symbol table /boot/System.map-4.9.78-yocto-standard unavailable, (No 
such file or directory)
+WARNING: Kernel function symbol table missing [man warning::symbols]
+
+semantic error: while resolving probe point: identifier 'kernel' at 
meminfo.stp:1:7
+   thrown from: ../systemtap/elaborate.cxx:1080
+source: probe kernel.function("meminfo_proc_show") {
+  ^
+
+semantic error: missing x86_64 kernel/module debuginfo [man 
warning::debuginfo] under 
'/home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs//lib/modules/4.9.78-yocto-standard/build'
+   thrown from: ../systemtap/dwflpp.cxx:340
+
+Signed-off-by: Victor Kamensky <kamen...@cisco.com>
+---
+ session.cxx | 2 +-
+ 1 file changed, 1 insertion(+), 1 deletion(-)
+
+diff --git a/session.cxx b/session.cxx
+index c7eaf6f..37caf3f 100644
+--- a/session.cxx
 b/session.cxx
+@@ -2038,7 +2038,7 @@ systemtap_session::parse_kernel_f

Re: [OE-core] [PATCH v2 1/2] systemtap: supports mips starting with version 3.1

[Victor Kamensky]

Hi Khem,

On Mon, 5 Mar 2018, Khem Raj wrote:


On Mon, Mar 5, 2018 at 1:50 PM, Victor Kamensky <kamen...@cisco.com> wrote:

Signed-off-by: Victor Kamensky <kamen...@cisco.com>
---
 meta/recipes-kernel/systemtap/systemtap_git.inc | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/meta/recipes-kernel/systemtap/systemtap_git.inc 
b/meta/recipes-kernel/systemtap/systemtap_git.inc
index c938760..d122359 100644
--- a/meta/recipes-kernel/systemtap/systemtap_git.inc
+++ b/meta/recipes-kernel/systemtap/systemtap_git.inc
@@ -15,8 +15,7 @@ SRC_URI = "git://sourceware.org/git/systemtap.git \

file://0001-staprun-stapbpf-don-t-support-installing-a-non-root.patch \
"

-# systemtap doesn't support mips
-COMPATIBLE_HOST = '(x86_64|i.86|powerpc|arm|aarch64|microblazeel).*-linux'
+COMPATIBLE_HOST = '(x86_64|i.86|powerpc|arm|aarch64|microblazeel|mips).*-linux'



Looks ok. Can we enable mipsel as well ?


Above expression, effectively 'mips.*-linux', does match 'mipsel' as well, 
does not it? It matches mips64.


I did test basic systemtap operations on qemumips, and
qemumips64 machines, worked ok to me. I noticed some small quirks 
but they were similar to one that arm machine has. I am not saying

that MIPS support in systemtap is perfect, it is new architecture
for systemtap after all, there could be some bugs. But I believe
it is good enough to be enabled in OE.

I don't see any easy virtual mipsel based machine so I cannot test
it right away. But I don't see a reson why it would not work in LE
mode.

Thanks,
Victor


 S = "${WORKDIR}/git"

--
2.7.4

--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core



--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH v2 2/2] systemtap: bring in PR22551 fix that addresses build against 4.15 kernel issue

[Victor Kamensky]

With 4.15 kernel systemtap needs update to address systemtap module
compilation issues. It is fixed in later version of systemtap by
PR22551.

Upstream-Status: Backport
Signed-off-by: Victor Kamensky <kamen...@cisco.com>
---
 ...-by-updating-the-use-of-timers-for-the-4..patch | 277 +
 meta/recipes-kernel/systemtap/systemtap_git.inc|   1 +
 2 files changed, 278 insertions(+)
 create mode 100644 
meta/recipes-kernel/systemtap/systemtap/0001-Fix-PR22551-by-updating-the-use-of-timers-for-the-4..patch

diff --git 
a/meta/recipes-kernel/systemtap/systemtap/0001-Fix-PR22551-by-updating-the-use-of-timers-for-the-4..patch
 
b/meta/recipes-kernel/systemtap/systemtap/0001-Fix-PR22551-by-updating-the-use-of-timers-for-the-4..patch
new file mode 100644
index 000..460c3b7
--- /dev/null
+++ 
b/meta/recipes-kernel/systemtap/systemtap/0001-Fix-PR22551-by-updating-the-use-of-timers-for-the-4..patch
@@ -0,0 +1,277 @@
+From fbb26e17a4c026f05a497fc5d584516bad3b6950 Mon Sep 17 00:00:00 2001
+From: David Smith <dsm...@redhat.com>
+Date: Wed, 6 Dec 2017 14:37:42 -0600
+Subject: [PATCH] Fix PR22551 by updating the use of timers for the 4.15
+ kernel.
+
+* runtime/linux/timer_compatibility.h: New file.
+* runtime/time.c: Update timer callback function parameter type. Update
+  timer initialization.
+* runtime/transport/relay_v2.c: Ditto.
+* runtime/transport/transport.c: Ditto.
+* tapset-timers.cxx (timer_derived_probe_group::emit_module_decls):
+  Ditto. Handle old and new timer callback interface.
+* runtime/linux/runtime.h: Include timer_compatibility.h instead of timer.h.
+* tapset/linux/scsi.stp: Ditto.
+
+Upstream-Status: Backport
+Signed-off-by: Victor Kamensky <kamen...@cisco.com>
+
+---
+ runtime/linux/runtime.h |  2 +-
+ runtime/linux/timer_compatibility.h | 76 +
+ runtime/time.c  |  7 ++--
+ runtime/transport/relay_v2.c|  8 ++--
+ runtime/transport/transport.c   | 13 +++
+ tapset-timers.cxx   | 14 +--
+ tapset/linux/scsi.stp   |  2 +-
+ 7 files changed, 100 insertions(+), 22 deletions(-)
+ create mode 100644 runtime/linux/timer_compatibility.h
+
+diff --git a/runtime/linux/runtime.h b/runtime/linux/runtime.h
+index 9c585a2..df9b74c 100644
+--- a/runtime/linux/runtime.h
 b/runtime/linux/runtime.h
+@@ -34,7 +34,7 @@
+ #include 
+ #include 
+ #include 
+-#include 
++#include "timer_compatibility.h"
+ #include 
+ #include 
+ #include 
+diff --git a/runtime/linux/timer_compatibility.h 
b/runtime/linux/timer_compatibility.h
+new file mode 100644
+index 000..ac03de9
+--- /dev/null
 b/runtime/linux/timer_compatibility.h
+@@ -0,0 +1,76 @@
++/*
++ * linux/timer.h compatibility defines and inlines
++ * Copyright (C) 2017 Red Hat Inc.
++ *
++ * This file is part of systemtap, and is free software.  You can
++ * redistribute it and/or modify it under the terms of the GNU General
++ * Public License (GPL); either version 2, or (at your option) any
++ * later version.
++ */
++
++#ifndef _TIMER_COMPATIBILITY_H_
++#define _TIMER_COMPATIBILITY_H_
++
++#include 
++
++/*
++ * Starting with the 4.15 kernel, the timer interface
++ * changed. Originally, you'd do something like:
++ *
++ *   static void timer_func(unsigned long val);
++ *
++ *   init_timer();
++ *   timer.expires = jiffies + STP_RELAY_TIMER_INTERVAL;
++ *   timer.function = timer_func;
++ *   timer.data = 0;
++ *   add_timer();
++ *
++ * The 'data' parameter would get passed to the callback
++ * function. Starting with 4.15, you'd do something like this:
++ *
++ *   static void timer_func(struct timer_list *val);
++ *
++ *   timer_setup(, timer_func, 0);
++ *   timer.expires = jiffies + STP_RELAY_TIMER_INTERVAL;
++ *   add_timer();
++ *   
++ * With the new code, the timer that caused the callback gets passed
++ * to the timer callback function. The 'data' field has been removed.
++ *
++ * So, we're going to use the new interface. To hide the differences
++ * between the callback function parameter type, we'll define a new
++ * type, 'stp_timer_callback_parameter_t'.
++ *
++ * If code needs to figure out the difference between the old and new
++ * interface, it should test the TIMER_TRACE_FLAGMASK define (which
++ * only exists in the new interface).
++ */
++
++#if defined(TIMER_TRACE_FLAGMASK) 
++/* This is the >= 4.15 kernel interface. */
++
++typedef struct timer_list * stp_timer_callback_parameter_t;
++
++#else
++/* This is the < 4.15 kernel interface. */
++
++typedef unsigned long stp_timer_callback_parameter_t;
++
++/**
++ * timer_setup - prepare a timer for first use
++ * @timer: the timer in question
++ * @callback: the function to call when timer expires
++ * @flags: any TIMER_* flags (note that anything other than 0 is an
++ *   error, since this compatibility function can't support any
++ *   of the TIMER_* flags)
++ */
++#define 

[OE-core] [PATCH v2 1/2] systemtap: supports mips starting with version 3.1

[Victor Kamensky]

Signed-off-by: Victor Kamensky <kamen...@cisco.com>
---
 meta/recipes-kernel/systemtap/systemtap_git.inc | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/meta/recipes-kernel/systemtap/systemtap_git.inc 
b/meta/recipes-kernel/systemtap/systemtap_git.inc
index c938760..d122359 100644
--- a/meta/recipes-kernel/systemtap/systemtap_git.inc
+++ b/meta/recipes-kernel/systemtap/systemtap_git.inc
@@ -15,8 +15,7 @@ SRC_URI = "git://sourceware.org/git/systemtap.git \

file://0001-staprun-stapbpf-don-t-support-installing-a-non-root.patch \
"
 
-# systemtap doesn't support mips
-COMPATIBLE_HOST = '(x86_64|i.86|powerpc|arm|aarch64|microblazeel).*-linux'
+COMPATIBLE_HOST = '(x86_64|i.86|powerpc|arm|aarch64|microblazeel|mips).*-linux'
 
 S = "${WORKDIR}/git"
 
-- 
2.7.4

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH 2/2] systemtap: bring in PR22551 fix that addresses build against 4.15 kernel issue

[Victor Kamensky]

With 4.15 kernel systemtap needs update to address systemtap module
compilation issues. It is fixed in later version of systemtap by
PR22551.

Upstream-Status: Backport

Signed-off-by: Victor Kamensky <kamen...@cisco.com>
---
 ...-by-updating-the-use-of-timers-for-the-4..patch | 273 +
 meta/recipes-kernel/systemtap/systemtap_git.inc|   1 +
 2 files changed, 274 insertions(+)
 create mode 100644 
meta/recipes-kernel/systemtap/systemtap/0001-Fix-PR22551-by-updating-the-use-of-timers-for-the-4..patch

diff --git 
a/meta/recipes-kernel/systemtap/systemtap/0001-Fix-PR22551-by-updating-the-use-of-timers-for-the-4..patch
 
b/meta/recipes-kernel/systemtap/systemtap/0001-Fix-PR22551-by-updating-the-use-of-timers-for-the-4..patch
new file mode 100644
index 000..da30f8c
--- /dev/null
+++ 
b/meta/recipes-kernel/systemtap/systemtap/0001-Fix-PR22551-by-updating-the-use-of-timers-for-the-4..patch
@@ -0,0 +1,273 @@
+From fbb26e17a4c026f05a497fc5d584516bad3b6950 Mon Sep 17 00:00:00 2001
+From: David Smith <dsm...@redhat.com>
+Date: Wed, 6 Dec 2017 14:37:42 -0600
+Subject: [PATCH] Fix PR22551 by updating the use of timers for the 4.15
+ kernel.
+
+* runtime/linux/timer_compatibility.h: New file.
+* runtime/time.c: Update timer callback function parameter type. Update
+  timer initialization.
+* runtime/transport/relay_v2.c: Ditto.
+* runtime/transport/transport.c: Ditto.
+* tapset-timers.cxx (timer_derived_probe_group::emit_module_decls):
+  Ditto. Handle old and new timer callback interface.
+* runtime/linux/runtime.h: Include timer_compatibility.h instead of timer.h.
+* tapset/linux/scsi.stp: Ditto.
+---
+ runtime/linux/runtime.h |  2 +-
+ runtime/linux/timer_compatibility.h | 76 +
+ runtime/time.c  |  7 ++--
+ runtime/transport/relay_v2.c|  8 ++--
+ runtime/transport/transport.c   | 13 +++
+ tapset-timers.cxx   | 14 +--
+ tapset/linux/scsi.stp   |  2 +-
+ 7 files changed, 100 insertions(+), 22 deletions(-)
+ create mode 100644 runtime/linux/timer_compatibility.h
+
+diff --git a/runtime/linux/runtime.h b/runtime/linux/runtime.h
+index 9c585a2..df9b74c 100644
+--- a/runtime/linux/runtime.h
 b/runtime/linux/runtime.h
+@@ -34,7 +34,7 @@
+ #include 
+ #include 
+ #include 
+-#include 
++#include "timer_compatibility.h"
+ #include 
+ #include 
+ #include 
+diff --git a/runtime/linux/timer_compatibility.h 
b/runtime/linux/timer_compatibility.h
+new file mode 100644
+index 000..ac03de9
+--- /dev/null
 b/runtime/linux/timer_compatibility.h
+@@ -0,0 +1,76 @@
++/*
++ * linux/timer.h compatibility defines and inlines
++ * Copyright (C) 2017 Red Hat Inc.
++ *
++ * This file is part of systemtap, and is free software.  You can
++ * redistribute it and/or modify it under the terms of the GNU General
++ * Public License (GPL); either version 2, or (at your option) any
++ * later version.
++ */
++
++#ifndef _TIMER_COMPATIBILITY_H_
++#define _TIMER_COMPATIBILITY_H_
++
++#include 
++
++/*
++ * Starting with the 4.15 kernel, the timer interface
++ * changed. Originally, you'd do something like:
++ *
++ *   static void timer_func(unsigned long val);
++ *
++ *   init_timer();
++ *   timer.expires = jiffies + STP_RELAY_TIMER_INTERVAL;
++ *   timer.function = timer_func;
++ *   timer.data = 0;
++ *   add_timer();
++ *
++ * The 'data' parameter would get passed to the callback
++ * function. Starting with 4.15, you'd do something like this:
++ *
++ *   static void timer_func(struct timer_list *val);
++ *
++ *   timer_setup(, timer_func, 0);
++ *   timer.expires = jiffies + STP_RELAY_TIMER_INTERVAL;
++ *   add_timer();
++ *   
++ * With the new code, the timer that caused the callback gets passed
++ * to the timer callback function. The 'data' field has been removed.
++ *
++ * So, we're going to use the new interface. To hide the differences
++ * between the callback function parameter type, we'll define a new
++ * type, 'stp_timer_callback_parameter_t'.
++ *
++ * If code needs to figure out the difference between the old and new
++ * interface, it should test the TIMER_TRACE_FLAGMASK define (which
++ * only exists in the new interface).
++ */
++
++#if defined(TIMER_TRACE_FLAGMASK) 
++/* This is the >= 4.15 kernel interface. */
++
++typedef struct timer_list * stp_timer_callback_parameter_t;
++
++#else
++/* This is the < 4.15 kernel interface. */
++
++typedef unsigned long stp_timer_callback_parameter_t;
++
++/**
++ * timer_setup - prepare a timer for first use
++ * @timer: the timer in question
++ * @callback: the function to call when timer expires
++ * @flags: any TIMER_* flags (note that anything other than 0 is an
++ *   error, since this compatibility function can't support any
++ *   of the TIMER_* flags)
++ */
++#define timer_setup(timer, callback, flags)   \
++  {   \
++   

[OE-core] [PATCH 1/2] systemtap: supports mips starting with version 3.1

[Victor Kamensky]

Signed-off-by: Victor Kamensky <kamen...@cisco.com>
---
 meta/recipes-kernel/systemtap/systemtap_git.inc | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/meta/recipes-kernel/systemtap/systemtap_git.inc 
b/meta/recipes-kernel/systemtap/systemtap_git.inc
index c938760..d122359 100644
--- a/meta/recipes-kernel/systemtap/systemtap_git.inc
+++ b/meta/recipes-kernel/systemtap/systemtap_git.inc
@@ -15,8 +15,7 @@ SRC_URI = "git://sourceware.org/git/systemtap.git \

file://0001-staprun-stapbpf-don-t-support-installing-a-non-root.patch \
"
 
-# systemtap doesn't support mips
-COMPATIBLE_HOST = '(x86_64|i.86|powerpc|arm|aarch64|microblazeel).*-linux'
+COMPATIBLE_HOST = '(x86_64|i.86|powerpc|arm|aarch64|microblazeel|mips).*-linux'
 
 S = "${WORKDIR}/git"
 
-- 
2.7.4

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH 0/1] systemtap: fix build failure on host system with dyninst installed

[Victor Kamensky]

Building systemap on Fedora 18 got the following failure

DEBUG: Shell function do_configure finished
DEBUG: Executing python function do_qa_configure
NOTE: Checking autotools environment for common misconfiguration
ERROR: This autoconf log indicates errors, it looked at host include and/or 
library paths while determining system capabilities.
Rerun configure task after fixing this. The path was 
'/run/media/kamensky/wd/lng/080713/buildle/tmp-eglibc/work/armv7ahf-vfp-neon-oe-linux-gnueabi/systemtap/2.1+gitAUTOINC+addec813fe-r1/git'
DEBUG: Python function do_qa_configure finished
ERROR: Function failed: do_qa_configure

Inside of config log have the following cross compilation
issue:

configure:10017: arm-oe-linux-gnueabi-g++  -march=armv7-a -marm 
-mthumb-interwork -mfloat-abi=hard -mfpu=neon 
--sysroot=/run/media/kamensky/wd/lng/080713/buildle/tmp-eglibc/sysroots/genericarmv7a
 -c -O2 -pipe -g -feliminate-unused-debug-types -fpermissive 
-fvisibility-inlines-hidden -fstack-protector-all -D_FORTIFY_SOURCE=2  
-I/usr/include/dyninst conftest.cpp 5
cc1plus: warning: include location /usr/include/dyninst is unsafe for 
cross-compilation [-Wpoison-system-directories]
In file included from /usr/include/dyninst/dyntypes.h:158:0,
 from /usr/include/dyninst/BPatch_object.h:36,
 from conftest.cpp:74:
/usr/include/dyninst/dyn_regs.h:36:32: fatal error: boost/shared_ptr.hpp: No 
such file or directory
 #include boost/shared_ptr.hpp
^
compilation terminated.

For now simple fix is to disable dyninst with --without-dyninst.
Proposed patch follows this cover letter

Thanks,
Victor

Victor Kamensky (1):
  systemtap: fix build failure on host system with dyninst installed

 meta/recipes-kernel/systemtap/systemtap_git.bb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

-- 
1.8.1.4

___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH 1/1] systemtap: fix build failure on host system with dyninst installed

[Victor Kamensky]

On host system where dyninst installed (i.e Fedora 18 with SystemTap)
builds fails with the error from do_qa_configure This autoconf log
indicates errors, it looked at host include and/or library paths
while determining system capabilities. Problematic config.log
indicates inclusion of /usr/include/dyninst/dyntypes.h

Fix for now - disable dyninst in oe builds

Signed-off-by: Victor Kamensky victor.kamen...@linaro.org
---
 meta/recipes-kernel/systemtap/systemtap_git.bb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/recipes-kernel/systemtap/systemtap_git.bb 
b/meta/recipes-kernel/systemtap/systemtap_git.bb
index 9bba5a9..4f60f6b 100644
--- a/meta/recipes-kernel/systemtap/systemtap_git.bb
+++ b/meta/recipes-kernel/systemtap/systemtap_git.bb
@@ -13,7 +13,7 @@ RDEPENDS_${PN}_class-nativesdk += python-native
 PR = r1
 
 EXTRA_OECONF += --with-libelf=${STAGING_DIR_TARGET} --without-rpm \
---without-nss --without-avahi \
+--without-nss --without-avahi --without-dyninst \
 --disable-server --disable-grapher 
 
 STAP_DOCS ?= --disable-docs --disable-publican --disable-refdocs
-- 
1.8.1.4

___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core