Bug#962994: [pcp] Bug#962994: pcp: cron jobs launch pcp in cron's cgroup

[Nathan Scott]

Hi Sam,

On Wed, Jan 6, 2021 at 5:38 AM Sam Morris  wrote:
> [...]
>
> checking if systemd should be used... no
>
> I wish it was possible to see the config.log from this build... but at
> least I can reproduce this with pbuilder.
>
> Adding --with-systemd to the configure command line will promote this to
> a build error. I recommend making that change in debian/rules so that
> this doesn't fall through the cracks:

Agreed - will do.

> [...]
> And it's systemd that ships systemd.pc:
>
> $ dpkg -S /usr/share/pkgconfig/systemd.pc
> systemd: /usr/share/pkgconfig/systemd.pc

Aha!  I get it now - the libsystem-dev dependency is insufficient, we
need a dependency on systemd itself (as you say).  I'll get that fixed
up too.  Thanks!!!

cheers.

--
Nathan

Bug#962994: pcp: cron jobs launch pcp in cron's cgroup

[Sam Morris]

Package: pcp
Version: 5.2.3-1
Followup-For: Bug #962994

This doesn't appear to be fixed. I can see init scripts & cron job
definitions but not systemd units present in the pcp package:

$ dpkg -L pcp | grep systemd
/usr/lib/pcp/pmdas/systemd
/usr/lib/pcp/pmdas/systemd/Install
/usr/lib/pcp/pmdas/systemd/README
/usr/lib/pcp/pmdas/systemd/Remove
/usr/lib/pcp/pmdas/systemd/domain.h
/usr/lib/pcp/pmdas/systemd/help
/usr/lib/pcp/pmdas/systemd/pmdasystemd
/usr/lib/pcp/pmdas/systemd/pmns
/usr/lib/pcp/pmdas/systemd/root
/usr/share/man/man1/pmdasystemd.1.gz
/var/lib/pcp/pmdas/systemd
/var/lib/pcp/pmdas/systemd/Install
/var/lib/pcp/pmdas/systemd/README
/var/lib/pcp/pmdas/systemd/Remove
/var/lib/pcp/pmdas/systemd/domain.h
/var/lib/pcp/pmdas/systemd/help
/var/lib/pcp/pmdas/systemd/pmdasystemd
/var/lib/pcp/pmdas/systemd/pmns
/var/lib/pcp/pmdas/systemd/root

In 

we still have:

checking if systemd should be used... no

I wish it was possible to see the config.log from this build... but at
least I can reproduce this with pbuilder.

Adding --with-systemd to the configure command line will promote this to
a build error. I recommend making that change in debian/rules so that
this doesn't fall through the cracks:

--- debian/rules.old2021-01-04 11:25:11.569164314 +
+++ debian/rules2021-01-04 11:08:33.345991196 +
@@ -121,7 +121,7 @@
 #   ... Makepkgs uses the latter mechanism to refine the configure
 #   options
 #
-configure_paths = --prefix=/usr --libexecdir=/usr/lib --sysconfdir=/etc 
--localstatedir=/var --with-rcdir=/etc/init.d --with-sysconfigdir=/etc/default
+configure_paths = --prefix=/usr --libexecdir=/usr/lib --sysconfdir=/etc 
--localstatedir=/var --with-rcdir=/etc/init.d --with-sysconfigdir=/etc/default 
--with-systemd

 checkdir = test -f debian/rules
 uninstall = cat debian/*.install | sed -e "s,^,debian/$(pcp)/," | xargs rm 
-f

With that in place, I get:

checking if systemd should be used... configure: error: cannot enable 
systemd support - no systemunitdir path
make: *** [debian/rules:150: .census] Error 1
dpkg-buildpackage: error: debian/rules build subprocess returned exit 
status 2
I: copying local configuration
E: Failed autobuilding of package

This is because there is no build-dependency on systemd:

root@fragarach:/build/pcp-5.2.3# dpkg -l systemd
dpkg-query: no packages found matching systemd

And it's systemd that ships systemd.pc:

$ dpkg -S /usr/share/pkgconfig/systemd.pc
systemd: /usr/share/pkgconfig/systemd.pc

I got the build working:

checking if systemd should be used... yes
checking systemd/sd-daemon.h usability... yes
checking systemd/sd-daemon.h presence... yes
checking for systemd/sd-daemon.h... yes

... by adding the build-dependency:

$ diff -u debian/control.old debian/control
--- debian/control.old  2021-01-04 11:24:51.005091436 +
+++ debian/control  2021-01-04 11:25:19.965194070 +
@@ -4,7 +4,7 @@
 Homepage: https://pcp.io
 Maintainer: PCP Development Team 
 Uploaders: Nathan Scott , Ken McDonell 

-Build-Depends: bison, flex, gawk, procps, pkg-config, debhelper (>= 5), 
perl (>= 5.6), libreadline-dev | libreadline5-dev | libreadline-gplv2-dev, 
chrpath, libbsd-dev [kfreebsd-any], libkvm-dev [kfreebsd-any], python3-dev, 
libnspr4-dev, libnss3-dev, libsasl2-dev, libuv1-dev, libssl-dev, 
libavahi-common-dev, qtbase5-dev, qtbase5-dev-tools, libqt5svg5-dev, qtchooser, 
autotools-dev, zlib1g-dev, autoconf, libclass-dbi-perl, libdbd-mysql-perl, 
python3-psycopg2, dh-python, libpfm4-dev, libncurses5-dev, python3-six, 
python3-json-pointer, python3-requests, libextutils-autoinstall-perl, 
libxml-tokeparser-perl, librrds-perl, libjson-perl, libwww-perl, 
libnet-snmp-perl, libnss3-tools, liblzma-dev, libsystemd-dev, bpftrace (>= 
0.9.2) [amd64 arm64 ppc64el], libibumad-dev, libibmad-dev, manpages, systemd
+Build-Depends: bison, flex, gawk, procps, pkg-config, debhelper (>= 5), 
perl (>= 5.6), libreadline-dev | libreadline5-dev | libreadline-gplv2-dev, 
chrpath, libbsd-dev [kfreebsd-any], libkvm-dev [kfreebsd-any], python3-dev, 
libnspr4-dev, libnss3-dev, libsasl2-dev, libuv1-dev, libssl-dev, 
libavahi-common-dev, qtbase5-dev, qtbase5-dev-tools, libqt5svg5-dev, qtchooser, 
autotools-dev, zlib1g-dev, autoconf, libclass-dbi-perl, libdbd-mysql-perl, 
python3-psycopg2, dh-python, libpfm4-dev, libncurses5-dev, python3-six, 
python3-json-pointer, python3-requests, libextutils-autoinstall-perl, 
libxml-tokeparser-perl, librrds-perl, libjson-perl, libwww-perl, 
libnet-snmp-perl, libnss3-tools, liblzma-dev, libsystemd-dev, bpftrace (>= 
0.9.2) [amd64 arm64 ppc64el], libibumad-dev, libibmad-dev, manpages
 Standards-Version: 3.9.3
 X-Python3-Version: >= 3.3
 
(I guess 

Bug#962994: [pcp] Bug#962994: pcp: cron jobs launch pcp in cron's cgroup

[Nathan Scott]

On Wed, Jun 17, 2020 at 9:52 PM Ken McDonell  wrote:
> On 17 June 2020 8:45:12 pm Sam Morris  wrote:
>
> > Package: pcp
> > Version: 5.1.1-1
> > Severity: normal
> >
> > $ systemctl status cron
> > ● cron.service - Regular background program processing daemon
> > Loaded: loaded (/lib/systemd/system/cron.service; enabled; vendor 
> > preset:
> > enabled)
> > Active: active (running) since Wed 2020-06-17 09:03:18 BST; 2min 5s ago
> >   Docs: man:cron(8)
> >   Main PID: 3499535 (cron)
> > IP: 0B in, 0B out
> >  Tasks: 49 (limit: 38380)
> > Memory: 2.4G
> >CPU: 728ms
> > CGroup: /system.slice/cron.service
> > ├─2725001 /usr/bin/pmie -b -F -P -l
> > /var/log/pcp/pmie/fragarach.domain.example/pmie.log -c 
> > config.default
> > ├─3241662 /usr/lib/pcp/bin/pmlogger -N -P -r -T24h10m -c 
> > config.default -v
> > 100mb -m pmlogger_check %Y%m%d.%H.%M
> > ...
>
> Hi Sam,
>
> Do you expect all these pmloggers to be running?
>
> We're going to need some help to debug this one.
>

I found some clues.  Firstly, and most obviously I guess - we should
not even be installing these cron jobs - this task is managed by our
systemd timers now.

Digging into that aspect, it seems the PCP configure script has now
started disabling systemd (!) on Debian for reasons I don't follow.

Here's a relevant build log:
https://buildd.debian.org/status/fetch.php?pkg=pcp=amd64=5.1.1-1=1590715276=0

This is the relevant PCP configure.ac snippet:

dnl Check for systemd services
enable_systemd=false
AC_MSG_CHECKING([if systemd should be used])
AS_IF([test "x$do_systemd" != "xno"], [
enable_systemd=true

PKG_CHECK_VAR([SYSTEMD_SYSTEMUNITDIR], [systemd], [systemdsystemunitdir],
[pcp_systemdunit_dir=$SYSTEMD_SYSTEMUNITDIR], [enable_systemd=false])

so, we are (all of a sudden?) taking the enable_systemd=false path
and looks like it's all downhill from there.

cheers.

--
Nathan

Bug#962994: [pcp] Bug#962994: pcp: cron jobs launch pcp in cron's cgroup

[Ken McDonell]

On 17 June 2020 8:45:12 pm Sam Morris  wrote:


Package: pcp
Version: 5.1.1-1
Severity: normal

$ systemctl status cron
● cron.service - Regular background program processing daemon
Loaded: loaded (/lib/systemd/system/cron.service; enabled; vendor preset: 
enabled)

Active: active (running) since Wed 2020-06-17 09:03:18 BST; 2min 5s ago
  Docs: man:cron(8)
  Main PID: 3499535 (cron)
IP: 0B in, 0B out
 Tasks: 49 (limit: 38380)
Memory: 2.4G
   CPU: 728ms
CGroup: /system.slice/cron.service
├─2725001 /usr/bin/pmie -b -F -P -l 
/var/log/pcp/pmie/fragarach.domain.example/pmie.log -c config.default
├─3241662 /usr/lib/pcp/bin/pmlogger -N -P -r -T24h10m -c config.default -v 
100mb -m pmlogger_check %Y%m%d.%H.%M

...


Hi Sam,

Do you expect all these pmloggers to be running?

We're going to need some help to debug this one.

Let's start with.

1. The contents of /etc/PCP/pmlogger ... tarball would be best
2. Output from $ sudo systemctl status pmlogger
3. Output from $ sudo systemctl status pmlogger_check
4. The contents of /var/log/pcp/pmlogger/pmlogger_daily.log* (maybe 1 or 
2 files)

5. The contents of /etc/cron.d/pcp* (maybe 0, 1 or more files)

Also, was this a fresh install of pcp-5.1.1 or did you upgrade (and if 
upgrade, do you know what the previous pcp version was)?


Fell free to respond to me directly if you don't want to spam the PCP 
list and Debian bug tracking system.

Bug#962994: [pcp] Bug#962994: pcp: cron jobs launch pcp in cron's cgroup

[Sam Morris]

On Wed, Jun 17, 2020 at 10:06:39PM +1000, Nathan Scott wrote:
> Digging into that aspect, it seems the PCP configure script has now
> started disabling systemd (!) on Debian for reasons I don't follow.
> 
> Here's a relevant build log:
> https://buildd.debian.org/status/fetch.php?pkg=pcp=amd64=5.1.1-1=1590715276=0
> 
> This is the relevant PCP configure.ac snippet:
> 
> dnl Check for systemd services
> enable_systemd=false
> AC_MSG_CHECKING([if systemd should be used])
> AS_IF([test "x$do_systemd" != "xno"], [
> enable_systemd=true
> 
> PKG_CHECK_VAR([SYSTEMD_SYSTEMUNITDIR], [systemd], [systemdsystemunitdir],
> [pcp_systemdunit_dir=$SYSTEMD_SYSTEMUNITDIR], [enable_systemd=false])
> 
> so, we are (all of a sudden?) taking the enable_systemd=false path
> and looks like it's all downhill from there.

I bet the buildd chroots don't have systemd installed, and you don't
build-depend on it. You probably also want to build-depend on
libsystemd-dev, which ships the sd-daemon.h header used by another check
in configure.ac.

More generally though, it's not great for a Debian package to ship timer
units vs cron jobs based on whether systemd was installed when the
package is built.

This is because some users still prefer to use sysvinit on their Debian
system, in which case nothing will be launching pcp's timer units.

The good news is that it's easy to handle this case by modifying your
cron jobs, for instance:

# Look for and purge old sessions every 30 minutes
09,39 * * * * root   [ -x /usr/lib/php/sessionclean ] && if [ ! -d 
/run/systemd/system ]; then /usr/lib/php/sessionclean; fi

It's also a good idea to check whether the program your cron job
launches is still runnable in order to handle the case where the package
has been removed (which removes the program) but not purged. In this
case, conffiles such as cron jobs will still be present, and root will
receive lots of mail from cron about the missing program. :)

-- 
Sam Morris 
CAAA AA1A CA69 A83A 892B  1855 D20B 4202 5CDA 27B9

Bug#962994: pcp: cron jobs launch pcp in cron's cgroup

[Sam Morris]

Package: pcp
Version: 5.1.1-1
Severity: normal

$ systemctl status cron
● cron.service - Regular background program processing daemon
 Loaded: loaded (/lib/systemd/system/cron.service; enabled; vendor preset: 
enabled)
 Active: active (running) since Wed 2020-06-17 09:03:18 BST; 2min 5s ago
   Docs: man:cron(8)
   Main PID: 3499535 (cron)
 IP: 0B in, 0B out
  Tasks: 49 (limit: 38380)
 Memory: 2.4G
CPU: 728ms
 CGroup: /system.slice/cron.service
 ├─2725001 /usr/bin/pmie -b -F -P -l 
/var/log/pcp/pmie/fragarach.domain.example/pmie.log -c config.default
 ├─3241662 /usr/lib/pcp/bin/pmlogger -N -P -r -T24h10m -c 
config.default -v 100mb -m pmlogger_check %Y%m%d.%H.%M
 ├─3244857 /usr/lib/pcp/bin/pmlogger -N -P -r -T24h10m -c 
config.default -v 100mb -mreexec %Y%m%d.%H.%M
 ├─3249634 /usr/lib/pcp/bin/pmlogger -N -P -r -T24h10m -c 
config.default -v 100mb -m pmlogger_check %Y%m%d.%H.%M
 ├─3253479 /usr/lib/pcp/bin/pmlogger -N -P -r -T24h10m -c 
config.default -v 100mb -m pmlogger_check %Y%m%d.%H.%M
 ├─3258915 /usr/lib/pcp/bin/pmlogger -N -P -r -T24h10m -c 
config.default -v 100mb -m pmlogger_check %Y%m%d.%H.%M
 ├─3262983 /usr/lib/pcp/bin/pmlogger -N -P -r -T24h10m -c 
config.default -v 100mb -m pmlogger_check %Y%m%d.%H.%M
 ├─3265468 /usr/lib/pcp/bin/pmlogger -N -P -r -T24h10m -c 
config.default -v 100mb -m pmlogger_check %Y%m%d.%H.%M
 ├─3267625 /usr/lib/pcp/bin/pmlogger -N -P -r -T24h10m -c 
config.default -v 100mb -m pmlogger_check %Y%m%d.%H.%M
 ├─3270077 /usr/lib/pcp/bin/pmlogger -N -P -r -T24h10m -c 
config.default -v 100mb -m pmlogger_check %Y%m%d.%H.%M
 ├─3276784 /usr/lib/pcp/bin/pmlogger -N -P -r -T24h10m -c 
config.default -v 100mb -m pmlogger_check %Y%m%d.%H.%M
 ├─3289498 /usr/lib/pcp/bin/pmlogger -N -P -r -T24h10m -c 
config.default -v 100mb -m pmlogger_check %Y%m%d.%H.%M
 ├─3314737 /usr/lib/pcp/bin/pmlogger -N -P -r -T24h10m -c 
config.default -v 100mb -m pmlogger_check %Y%m%d.%H.%M
 ├─3329439 /usr/lib/pcp/bin/pmlogger -N -P -r -T24h10m -c 
config.default -v 100mb -m pmlogger_check %Y%m%d.%H.%M
 ├─3339935 /usr/lib/pcp/bin/pmlogger -N -P -r -T24h10m -c 
config.default -v 100mb -m pmlogger_check %Y%m%d.%H.%M
 ├─3350479 /usr/lib/pcp/bin/pmlogger -N -P -r -T24h10m -c 
config.default -v 100mb -m pmlogger_check %Y%m%d.%H.%M
 ├─3360898 /usr/lib/pcp/bin/pmlogger -N -P -r -T24h10m -c 
config.default -v 100mb -m pmlogger_check %Y%m%d.%H.%M
 ├─3367195 /usr/lib/pcp/bin/pmlogger -N -P -r -T24h10m -c 
config.default -v 100mb -m pmlogger_check %Y%m%d.%H.%M
 ├─3384420 /usr/lib/pcp/bin/pmlogger -N -P -r -T24h10m -c 
config.default -v 100mb -m pmlogger_check %Y%m%d.%H.%M
 ├─3393600 /usr/lib/pcp/bin/pmlogger -N -P -r -T24h10m -c 
config.default -v 100mb -m pmlogger_check %Y%m%d.%H.%M
 ├─3411864 /usr/lib/pcp/bin/pmlogger -N -P -r -T24h10m -c 
config.default -v 100mb -m pmlogger_check %Y%m%d.%H.%M
 ├─3415294 /usr/lib/pcp/bin/pmlogger -N -P -r -T24h10m -c 
config.default -v 100mb -m pmlogger_check %Y%m%d.%H.%M
 ├─3417448 /usr/lib/pcp/bin/pmlogger -N -P -r -T24h10m -c 
config.default -v 100mb -m pmlogger_check %Y%m%d.%H.%M
 ├─3419785 /usr/lib/pcp/bin/pmlogger -N -P -r -T24h10m -c 
config.default -v 100mb -m pmlogger_check %Y%m%d.%H.%M
 ├─3440219 /usr/lib/pcp/bin/pmlogger -N -P -r -T24h10m -c 
config.default -v 100mb -m pmlogger_check %Y%m%d.%H.%M
 ├─3443004 /usr/lib/pcp/bin/pmlogger -N -P -r -T24h10m -c 
config.default -v 100mb -m pmlogger_check %Y%m%d.%H.%M
 ├─3451062 /usr/lib/pcp/bin/pmlogger -N -P -r -T24h10m -c 
config.default -v 100mb -m pmlogger_check %Y%m%d.%H.%M
 ├─3453158 /usr/lib/pcp/bin/pmlogger -N -P -r -T24h10m -c 
config.default -v 100mb -m pmlogger_check %Y%m%d.%H.%M
 ├─3455206 /usr/lib/pcp/bin/pmlogger -N -P -r -T24h10m -c 
config.default -v 100mb -m pmlogger_check %Y%m%d.%H.%M
 ├─3457327 /usr/lib/pcp/bin/pmlogger -N -P -r -T24h10m -c 
config.default -v 100mb -mreexec %Y%m%d.%H.%M
 ├─3460418 /usr/lib/pcp/bin/pmlogger -N -P -r -T24h10m -c 
config.default -v 100mb -m pmlogger_check %Y%m%d.%H.%M
 ├─3462492 /usr/lib/pcp/bin/pmlogger -N -P -r -T24h10m -c 
config.default -v 100mb -m pmlogger_check %Y%m%d.%H.%M
 ├─3464518 /usr/lib/pcp/bin/pmlogger -N -P -r -T24h10m -c 
config.default -v 100mb -m pmlogger_check %Y%m%d.%H.%M
 ├─3466563 /usr/lib/pcp/bin/pmlogger -N -P -r -T24h10m -c 
config.default -v 100mb -m pmlogger_check %Y%m%d.%H.%M
 ├─3468875 /usr/lib/pcp/bin/pmlogger -N -P -r -T24h10m -c 
config.default -v 100mb -m pmlogger_check %Y%m%d.%H.%M
 ├─3471006 /usr/lib/pcp/bin/pmlogger -N -P -r -T24h10m -c 
config.default -v 100mb -m pmlogger_check