Re: Lots of systemctl segfaults in Koji Rawhide

2020-05-20 Thread Adam Williamson 
On Wed, 2020-05-20 at 09:03 +0200, Petr Pisar wrote:
> On Tue, May 19, 2020 at 08:24:34PM -0700, Adam Williamson wrote:
> > On Tue, 2020-05-19 at 18:32 -0700, Adam Williamson wrote:
> > > The most suspicious change between the two build envs that I can see is
> > > openssl. GOOD has openssl-1.1.1g-1.fc33.x86_64 , and BAD has
> > > openssl-1.1.1g-2.fc33.x86_64 . I'm gonna try doing an openssl build
> > > with the patch from -2 reverted and see where that gets us.
> > 
> > OK, I think that did the trick. nbdkit's x86_64 build succeeded this
> > time, and the log shows no systemctl crashes:
> > 
> > https://kojipkgs.fedoraproject.org//work/tasks/7666/44717666/root.log
> > 
> > I'll file a bug on openssl to let tmraz know about the problem.
> > Hopefully between the two fixes, the next Rawhide may also be fixed...
> 
> Nice detective work. You could have save your work with Koschei.
> 
>  nbdkit history,
>  first failure with this test
> failure on x86_64:
> 
> supermin: exception: 
> Sys_error("/lib/modules/5.7.0-0.rc5.20200515git1ae7efb38854.1.fc33.x86_64/modules.dep:
>  No such file or directory")
> 
> with only 5 source packages changed and openssl going from 1:1.1.1g-1.fc33 to
> 1:1.1.1g-2.fc33 being one of them.

Good point! Somehow Koschei just didn't come to mind as something that
would be helpful here, but it would indeed have saved me a bit of time.
I did look for packages with frequent 'official' builds to try and find
one which would show a narrow window where the bug appeared, but of
course Koschei would've been the ideal place to look :)
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Lots of systemctl segfaults in Koji Rawhide

2020-05-20 Thread Richard W.M. Jones 
On Tue, May 19, 2020 at 08:43:01PM -0700, Adam Williamson wrote:
> On Tue, 2020-05-19 at 20:24 -0700, Adam Williamson wrote:
> > On Tue, 2020-05-19 at 18:32 -0700, Adam Williamson wrote:
> > > The most suspicious change between the two build envs that I can see is
> > > openssl. GOOD has openssl-1.1.1g-1.fc33.x86_64 , and BAD has
> > > openssl-1.1.1g-2.fc33.x86_64 . I'm gonna try doing an openssl build
> > > with the patch from -2 reverted and see where that gets us.
> > 
> > OK, I think that did the trick. nbdkit's x86_64 build succeeded this
> > time, and the log shows no systemctl crashes:
> > 
> > https://kojipkgs.fedoraproject.org//work/tasks/7666/44717666/root.log
> > 
> > I'll file a bug on openssl to let tmraz know about the problem.
> > Hopefully between the two fixes, the next Rawhide may also be fixed...
> 
> Bug reports:
> 
> openssl - https://bugzilla.redhat.com/show_bug.cgi?id=1837809
> libpcap - https://bugzilla.redhat.com/show_bug.cgi?id=1837812

Excellent detective work, thanks!

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
libguestfs lets you edit virtual machines.  Supports shell scripting,
bindings from many languages.  http://libguestfs.org
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Lots of systemctl segfaults in Koji Rawhide

2020-05-20 Thread Zbigniew Jędrzejewski-Szmek 
On Tue, May 19, 2020 at 04:52:55PM -0700, Adam Williamson wrote:
> In fact it seems to be a bit more complex than that, because systemd
> doesn't actually list its dependency on libpcap:
> 
> [adamw@adam libpcap (master)]$ rpm -q --requires systemd | grep pcap
> [adamw@adam libpcap (master)]$
> 
> but it must be encoded indirectly somehow, because if you try and
> remove libpcap, dnf refuses to because it'd remove systemd:
> 
> [adamw@adam libpcap (master)]$ sudo dnf remove libpcap
> [sudo] password for adamw: 
> Error: 
>  Problem: The operation would result in removing the following
> protected packages: systemd

systemd requires libip4tc.so.2()(64bit) provided by iptables-libs,
and iptables-libs requires libpcap.so.1()(64bit) provided by libpcap.

Zbyszek
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Lots of systemctl segfaults in Koji Rawhide

2020-05-20 Thread Zbigniew Jędrzejewski-Szmek 
On Tue, May 19, 2020 at 08:43:01PM -0700, Adam Williamson wrote:
> On Tue, 2020-05-19 at 20:24 -0700, Adam Williamson wrote:
> > On Tue, 2020-05-19 at 18:32 -0700, Adam Williamson wrote:
> > > The most suspicious change between the two build envs that I can see is
> > > openssl. GOOD has openssl-1.1.1g-1.fc33.x86_64 , and BAD has
> > > openssl-1.1.1g-2.fc33.x86_64 . I'm gonna try doing an openssl build
> > > with the patch from -2 reverted and see where that gets us.
> > 
> > OK, I think that did the trick. nbdkit's x86_64 build succeeded this
> > time, and the log shows no systemctl crashes:
> > 
> > https://kojipkgs.fedoraproject.org//work/tasks/7666/44717666/root.log
> > 
> > I'll file a bug on openssl to let tmraz know about the problem.
> > Hopefully between the two fixes, the next Rawhide may also be fixed...
> 
> Bug reports:
> 
> openssl - https://bugzilla.redhat.com/show_bug.cgi?id=1837809
> libpcap - https://bugzilla.redhat.com/show_bug.cgi?id=1837812

Adam, thanks for tracking this down!

Zbyszek
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Lots of systemctl segfaults in Koji Rawhide

2020-05-20 Thread Petr Pisar 
On Tue, May 19, 2020 at 08:24:34PM -0700, Adam Williamson wrote:
> On Tue, 2020-05-19 at 18:32 -0700, Adam Williamson wrote:
> > 
> > The most suspicious change between the two build envs that I can see is
> > openssl. GOOD has openssl-1.1.1g-1.fc33.x86_64 , and BAD has
> > openssl-1.1.1g-2.fc33.x86_64 . I'm gonna try doing an openssl build
> > with the patch from -2 reverted and see where that gets us.
> 
> OK, I think that did the trick. nbdkit's x86_64 build succeeded this
> time, and the log shows no systemctl crashes:
> 
> https://kojipkgs.fedoraproject.org//work/tasks/7666/44717666/root.log
> 
> I'll file a bug on openssl to let tmraz know about the problem.
> Hopefully between the two fixes, the next Rawhide may also be fixed...

Nice detective work. You could have save your work with Koschei.

 nbdkit history,
 first failure with this test
failure on x86_64:

supermin: exception: 
Sys_error("/lib/modules/5.7.0-0.rc5.20200515git1ae7efb38854.1.fc33.x86_64/modules.dep:
 No such file or directory")

with only 5 source packages changed and openssl going from 1:1.1.1g-1.fc33 to
1:1.1.1g-2.fc33 being one of them.

-- Petr


signature.asc
Description: PGP signature
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Lots of systemctl segfaults in Koji Rawhide

2020-05-19 Thread Adam Williamson 
On Tue, 2020-05-19 at 20:24 -0700, Adam Williamson wrote:
> On Tue, 2020-05-19 at 18:32 -0700, Adam Williamson wrote:
> > The most suspicious change between the two build envs that I can see is
> > openssl. GOOD has openssl-1.1.1g-1.fc33.x86_64 , and BAD has
> > openssl-1.1.1g-2.fc33.x86_64 . I'm gonna try doing an openssl build
> > with the patch from -2 reverted and see where that gets us.
> 
> OK, I think that did the trick. nbdkit's x86_64 build succeeded this
> time, and the log shows no systemctl crashes:
> 
> https://kojipkgs.fedoraproject.org//work/tasks/7666/44717666/root.log
> 
> I'll file a bug on openssl to let tmraz know about the problem.
> Hopefully between the two fixes, the next Rawhide may also be fixed...

Bug reports:

openssl - https://bugzilla.redhat.com/show_bug.cgi?id=1837809
libpcap - https://bugzilla.redhat.com/show_bug.cgi?id=1837812
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Lots of systemctl segfaults in Koji Rawhide

2020-05-19 Thread Adam Williamson 
On Tue, 2020-05-19 at 18:32 -0700, Adam Williamson wrote:
> 
> The most suspicious change between the two build envs that I can see is
> openssl. GOOD has openssl-1.1.1g-1.fc33.x86_64 , and BAD has
> openssl-1.1.1g-2.fc33.x86_64 . I'm gonna try doing an openssl build
> with the patch from -2 reverted and see where that gets us.

OK, I think that did the trick. nbdkit's x86_64 build succeeded this
time, and the log shows no systemctl crashes:

https://kojipkgs.fedoraproject.org//work/tasks/7666/44717666/root.log

I'll file a bug on openssl to let tmraz know about the problem.
Hopefully between the two fixes, the next Rawhide may also be fixed...
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Lots of systemctl segfaults in Koji Rawhide

2020-05-19 Thread Adam Williamson 
On Tue, 2020-05-19 at 17:19 -0700, Adam Williamson wrote:
> On Tue, 2020-05-19 at 16:34 -0700, Adam Williamson wrote:
> > On Tue, 2020-05-19 at 15:45 -0700, Adam Williamson wrote:
> > > still, having trouble pinning down a culprit; it's not kernel-5.7.0-
> > > 0.rc6.1.fc33 as the 20200518.n.0 compose failed, and that was run with
> > > the previous kernel build, which *succeeded* in the 20200517.n.0
> > > compose...
> > > 
> > > I guess we get to poke through everything built around the 17th and try
> > > to find a relevant change? :)
> > 
> > This should be *more or less* the list of candidate culprits, which I
> > just heath robinson'd out of Bodhi (yet again I wish we got compose
> > reports for doomed composes, it'd make this easier...). grub2 and
> > openssl are the ones that jump out at me right away...ceph is also in
> > there, and ceph is one of the things that *is* linked to libibverbs ,
> > though the change in ceph doesn't seem like a significant one...
> > 
> > aha! This looks juicy. systemctl is linked against libpcap.so.1 , part
> > of libpcap, which is in the list below, and this is the changelog for
> > it:
> > 
> > * Fri May 15 2020 Michal Ruprich  - 14:1.9.1-4
> > - Enabling rdma support in libpcap
> > 
> > the changelog date is May 15 but the pcap build actually ran on 2020-
> > 05-18, and indeed the last successful Rawhide compose (20200517.n.1)
> > had 1.9.1-3.fc33, and the first failed compose (20200518.n.0) has
> > 1.9.1-4.fc33.
> > 
> > I'm betting we have a circular dependency problem or something here...
> 
> Well, seems like I may be half right. We untagged libcap-1.9.1-4.fc33
> and I re-ran the nbdkit build. It seems that this has fixed the "error
> while loading shared libraries: libibverbs.so.1: cannot open shared
> object file: No such file or directory" errors...but not the segfaults
> :( we now just get segfaults all the way:
> 
> DEBUG util.py:602:  Running transaction
> DEBUG util.py:602:  /var/tmp/rpm-tmp.yHBGhZ: line 1: 1976807 Segmentation 
> fault  (core dumped) systemd-machine-id-setup &> /dev/null
> DEBUG util.py:602:  /var/tmp/rpm-tmp.yHBGhZ: line 21: 1976809 Segmentation 
> fault  (core dumped) systemctl daemon-reexec &> /dev/null
> DEBUG util.py:602:  /var/tmp/rpm-tmp.yHBGhZ: line 23: 1976811 Segmentation 
> fault  (core dumped) journalctl --update-catalog &> /dev/null
> DEBUG util.py:602:  /var/tmp/rpm-tmp.yHBGhZ: line 24: 1976813 Segmentation 
> fault  (core dumped) systemd-tmpfiles --create &> /dev/null
> DEBUG util.py:602:  /var/tmp/rpm-tmp.yHBGhZ: line 50: 1976821 Segmentation 
> fault  (core dumped) systemctl preset-all &> /dev/null
> DEBUG util.py:602:  /var/tmp/rpm-tmp.yHBGhZ: line 50: 1976823 Segmentation 
> fault  (core dumped) systemctl --global preset-all &> /dev/null
> DEBUG util.py:602:  /var/tmp/rpm-tmp.TL9iz0: line 6: 1976826 Segmentation 
> fault  (core dumped) /usr/bin/systemctl --no-reload preset dbus.socket
> 
> so there might be two different problems here, possibly?

OK, so I dug into the history of the segfaults a bit. Turns out they've
been going on for a few days. They only seem to happen on builds where
the systemd package gets pulled into the buildroot. The closest delta
I've got is this:

GOOD: https://koji.fedoraproject.org/koji/buildinfo?buildID=1507622 (Fri, 15 
May 2020 18:01:07 UTC)
BAD:  https://koji.fedoraproject.org/koji/buildinfo?buildID=1507650 (Fri, 15 
May 2020 19:49:31 UTC)

"GOOD" is the latest build I can find where systemd was pulled in as a
build dep (so the bug "should have" happened) and the bug didn't
happen. "BAD" is the earliest build I can find where the bug *did*
happen.

The most suspicious change between the two build envs that I can see is
openssl. GOOD has openssl-1.1.1g-1.fc33.x86_64 , and BAD has
openssl-1.1.1g-2.fc33.x86_64 . I'm gonna try doing an openssl build
with the patch from -2 reverted and see where that gets us.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Lots of systemctl segfaults in Koji Rawhide

2020-05-19 Thread Adam Williamson 
On Tue, 2020-05-19 at 16:34 -0700, Adam Williamson wrote:
> On Tue, 2020-05-19 at 15:45 -0700, Adam Williamson wrote:
> > still, having trouble pinning down a culprit; it's not kernel-5.7.0-
> > 0.rc6.1.fc33 as the 20200518.n.0 compose failed, and that was run with
> > the previous kernel build, which *succeeded* in the 20200517.n.0
> > compose...
> > 
> > I guess we get to poke through everything built around the 17th and try
> > to find a relevant change? :)
> 
> This should be *more or less* the list of candidate culprits, which I
> just heath robinson'd out of Bodhi (yet again I wish we got compose
> reports for doomed composes, it'd make this easier...). grub2 and
> openssl are the ones that jump out at me right away...ceph is also in
> there, and ceph is one of the things that *is* linked to libibverbs ,
> though the change in ceph doesn't seem like a significant one...
> 
> aha! This looks juicy. systemctl is linked against libpcap.so.1 , part
> of libpcap, which is in the list below, and this is the changelog for
> it:
> 
> * Fri May 15 2020 Michal Ruprich  - 14:1.9.1-4
> - Enabling rdma support in libpcap
> 
> the changelog date is May 15 but the pcap build actually ran on 2020-
> 05-18, and indeed the last successful Rawhide compose (20200517.n.1)
> had 1.9.1-3.fc33, and the first failed compose (20200518.n.0) has
> 1.9.1-4.fc33.
> 
> I'm betting we have a circular dependency problem or something here...

Well, seems like I may be half right. We untagged libcap-1.9.1-4.fc33
and I re-ran the nbdkit build. It seems that this has fixed the "error
while loading shared libraries: libibverbs.so.1: cannot open shared
object file: No such file or directory" errors...but not the segfaults
:( we now just get segfaults all the way:

DEBUG util.py:602:  Running transaction
DEBUG util.py:602:  /var/tmp/rpm-tmp.yHBGhZ: line 1: 1976807 Segmentation fault 
 (core dumped) systemd-machine-id-setup &> /dev/null
DEBUG util.py:602:  /var/tmp/rpm-tmp.yHBGhZ: line 21: 1976809 Segmentation 
fault  (core dumped) systemctl daemon-reexec &> /dev/null
DEBUG util.py:602:  /var/tmp/rpm-tmp.yHBGhZ: line 23: 1976811 Segmentation 
fault  (core dumped) journalctl --update-catalog &> /dev/null
DEBUG util.py:602:  /var/tmp/rpm-tmp.yHBGhZ: line 24: 1976813 Segmentation 
fault  (core dumped) systemd-tmpfiles --create &> /dev/null
DEBUG util.py:602:  /var/tmp/rpm-tmp.yHBGhZ: line 50: 1976821 Segmentation 
fault  (core dumped) systemctl preset-all &> /dev/null
DEBUG util.py:602:  /var/tmp/rpm-tmp.yHBGhZ: line 50: 1976823 Segmentation 
fault  (core dumped) systemctl --global preset-all &> /dev/null
DEBUG util.py:602:  /var/tmp/rpm-tmp.TL9iz0: line 6: 1976826 Segmentation fault 
 (core dumped) /usr/bin/systemctl --no-reload preset dbus.socket

so there might be two different problems here, possibly?
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Lots of systemctl segfaults in Koji Rawhide

2020-05-19 Thread Adam Williamson 
On Tue, 2020-05-19 at 16:34 -0700, Adam Williamson wrote:
> 
> aha! This looks juicy. systemctl is linked against libpcap.so.1 , part
> of libpcap, which is in the list below, and this is the changelog for
> it:
> 
> * Fri May 15 2020 Michal Ruprich  - 14:1.9.1-4
> - Enabling rdma support in libpcap
> 
> the changelog date is May 15 but the pcap build actually ran on 2020-
> 05-18, and indeed the last successful Rawhide compose (20200517.n.1)
> had 1.9.1-3.fc33, and the first failed compose (20200518.n.0) has
> 1.9.1-4.fc33.
> 
> I'm betting we have a circular dependency problem or something here...

Just to expand on this a bit, what I meant is, I bet adding a
libibverbs.so.0 dependency to libpcap gives dnf an irresolvable
dependency loop, because libibverbs depends on rdma-core and rdma-core
depends on quite a lot of other stuff, including libsystemd and
systemd. So now dnf is in a corner, because systemd needs libpcap which
(now) needs libibverbs which needs rdma-core which needs...systemd.

In fact it seems to be a bit more complex than that, because systemd
doesn't actually list its dependency on libpcap:

[adamw@adam libpcap (master)]$ rpm -q --requires systemd | grep pcap
[adamw@adam libpcap (master)]$

but it must be encoded indirectly somehow, because if you try and
remove libpcap, dnf refuses to because it'd remove systemd:

[adamw@adam libpcap (master)]$ sudo dnf remove libpcap
[sudo] password for adamw: 
Error: 
 Problem: The operation would result in removing the following
protected packages: systemd

anyhow, bottom line is, I think dnf winds up installing systemd and
then a bunch of other packages whose scriptlets call systemctl well
before libibverbs gets installed, and that's where all the failures
come from, as it tries to run systemctl (which is linked against
libpcap which is now linked against libibverbs) before libibverbs is
there.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Lots of systemctl segfaults in Koji Rawhide

2020-05-19 Thread Adam Williamson 
On Tue, 2020-05-19 at 15:45 -0700, Adam Williamson wrote:
> 
> still, having trouble pinning down a culprit; it's not kernel-5.7.0-
> 0.rc6.1.fc33 as the 20200518.n.0 compose failed, and that was run with
> the previous kernel build, which *succeeded* in the 20200517.n.0
> compose...
> 
> I guess we get to poke through everything built around the 17th and try
> to find a relevant change? :)

This should be *more or less* the list of candidate culprits, which I
just heath robinson'd out of Bodhi (yet again I wish we got compose
reports for doomed composes, it'd make this easier...). grub2 and
openssl are the ones that jump out at me right away...ceph is also in
there, and ceph is one of the things that *is* linked to libibverbs ,
though the change in ceph doesn't seem like a significant one...

aha! This looks juicy. systemctl is linked against libpcap.so.1 , part
of libpcap, which is in the list below, and this is the changelog for
it:

* Fri May 15 2020 Michal Ruprich  - 14:1.9.1-4
- Enabling rdma support in libpcap

the changelog date is May 15 but the pcap build actually ran on 2020-
05-18, and indeed the last successful Rawhide compose (20200517.n.1)
had 1.9.1-3.fc33, and the first failed compose (20200518.n.0) has
1.9.1-4.fc33.

I'm betting we have a circular dependency problem or something here...

awscli-1.18.61-1.fc33
python-boto3-1.13.11-1.fc33
khotkeys-5.18.90-1.fc33
python-botocore-1.16.11-1.fc33
plasma-workspace-wallpapers-5.18.90-1.fc33
cascadia-code-fonts-2005.15-1.fc33
breeze-gtk-5.18.90-1.fc33
plasma-pa-5.18.90-1.fc33
plasma-oxygen-5.18.90-1.fc33
plasma-vault-5.18.90-1.fc33
ksshaskpass-5.18.90-1.fc33
kwayland-integration-5.18.90-1.fc33
kscreenlocker-5.18.90-1.fc33
plymouth-theme-breeze-5.18.90-1.fc33
libsysstat-0.4.3-1.fc33
plasma-browser-integration-5.18.90-1.fc33
liblxqt-0.15.0-1.fc33
plasma-sdk-5.18.90-1.fc33
bluedevil-5.18.90-1.fc33
plasma-milou-5.18.90-1.fc33
kdecoration-5.18.90-1.fc33
polkit-kde-5.18.90-1.fc33
kactivitymanagerd-5.18.90-1.fc33
pam-kwallet-5.18.90-1.fc33
sddm-kcm-5.18.90-1.fc33
kgamma-5.18.90-1.fc33
libkscreen-qt5-5.18.90-1.fc33
grub2-breeze-theme-5.18.90-1.fc33
python-readme-renderer-26.0-2.fc33
kwrited-5.18.90-1.fc33
vim-8.2.789-1.fc33
phd2-2.6.8-1.fc33
libqtxdg-3.5.0-1.fc33
python-readme-renderer-26.0-1.fc33
grub2-2.04-20.fc33
perl-DateTime-Format-Natural-1.09-1.fc33
openssl-1.1.1g-3.fc33
rust-isahc-0.9.2-1.fc33
lxqt-build-tools-0.7.0-1.fc33
onedrive-2.4.1-1.fc33
rust-zincati-0.0.10-3.fc33
rust-afterburn-4.3.3-3.fc33
rust-reqwest-0.10.4-2.fc33
dummy-test-package-crested-0-512.fc33
dummy-test-package-rubino-0-512.fc33
java-11-openjdk-11.0.7.10-2.fc33
rust-procfs-0.7.9-1.fc33
rust-libflate_lz77-1.0.0-2.fc33
libyang-1.0.167-1.fc33
quota-4.05-12.fc33
libpcap-1.9.1-4.fc33
rust-libflate-1.0.0-1.fc33
rust-mockito-0.25.1-1.fc33
php-zmq-1.1.3-14.fc33
libpq-12.3-1.fc33
perl-Metrics-Any-0.05-1.fc33
rust-libflate_lz77-1.0.0-1.fc33
rust-testing_logger-0.1.1-1.fc33
ghostscript-9.52-2.fc33
ruby-2.7.1-131.fc33
rust-y4m-0.6.0-1.fc33
rust-datetime-0.5.0-2.fc33
licensecheck-3.0.47-1.fc33
mosquitto-1.6.9-4.fc33
rust-resize-0.4.0-1.fc33
rust-iso8601-0.4.0-1.fc33
rust-arbitrary-0.4.4-1.fc33
libwebp-1.1.0-3.fc33
perl-PkgConfig-LibPkgConf-0.11-1.fc33
iputils-20190515-7.fc33
perl-Alien-Build-2.23-1.fc33
dummy-test-package-gloster-0-538.fc33
xeus-0.23.14-1.fc33
python-rasterio-1.1.4-1.fc33
supertux-0.6.2-1.fc33
perl-Regexp-Pattern-License-3.3.1-1.fc33
kbibtex-0.9-6.fc33
unar-1.10.1-17.fc33
calibre-4.16.0-2.fc33
php-bartlett-php-compatinfo-db-2.14.0-1.fc33
libwebsockets-4.0.10-1.fc33
spill-0.8-21.fc33
memcached-1.6.6-1.fc33
dummy-test-package-crested-0-511.fc33
dummy-test-package-rubino-0-511.fc33
rust-quote-1.0.6-1.fc33
rust-mailparse-0.12.1-1.fc33
rubygem-rspec-expectations-3.9.2-1.fc33
libreoffice-6.4.3.2-2.fc33
python-cartopy-0.18.0-1.fc33
open-vm-tools-11.0.5-4.fc33
adobe-afdko-3.3.0-1.fc33
nuspell-3.1.1-1.fc33
dummy-test-package-gloster-0-537.fc33
grsync-1.2.8-1.fc33
sakura-3.7.1-1.fc33
bindfs-1.14.7-1.fc33
ghc-rpm-macros-2.0.8-1.fc33
redis-6.0.3-1.fc33
dummy-test-package-gloster-0-536.fc33
dummy-test-package-crested-0-509.fc33
dummy-test-package-rubino-0-509.fc33
python-scikit-learn-0.23.0-1.fc33
0ad-0.0.23b-14.fc33
389-ds-base-1.4.4.2-1.fc33.1
bes-3.20.6-3.fc33
boost-1.69.0-19.fc33
cfdg-3.3-2.fc33
community-mysql-8.0.20-2.fc33
cyrus-imapd-3.0.13-7.fc33
evolution-data-server-3.37.1-2.fc33
freerdp-2.0.0-57.20200207git245fc60.fc33
gnucash-3.10-2.fc33
harfbuzz-2.6.6-2.fc33
ibus-qt-1.3.3-25.fc33
icecat-68.8.0-4.rh1.fc33
icu-67.1-1.fc33
idzebra-2.1.4-8.fc33
imv-4.1.0-2.fc33
kiwix-lib-9.1.2-2.fc33
libcdr-0.1.6-2.fc33
libe-book-0.1.3-13.fc33
libical-3.0.8-2.fc33
libicu65-65.1-1.fc33
libmspub-0.1.4-12.fc33
libphonenumber-8.12.1-2.fc33
libqalculate-3.10.0-2.fc33
libqxp-0.0.2-7.fc33
libtoml-0-15.20161213git03e8a3a.fc33
libtranslit-0.0.3-27.fc33
libvisio-0.1.7-4.fc33
libzmf-0.0.2-13.fc33
mapnik-3.0.23-3.fc33
mongo-c-driver-1.17.0~beta-2.fc33
msort-8.53-36.fc33
ncmpcpp-0.8.2-14.fc33

Re: Lots of systemctl segfaults in Koji Rawhide

2020-05-19 Thread Doug Ledford 
On Tue, 2020-05-19 at 15:58 -0700, Adam Williamson wrote:
> On Tue, 2020-05-19 at 18:55 -0400, Doug Ledford wrote:
> > > I guess we get to poke through everything built around the 17th
> > > and
> > > try
> > > to find a relevant change? :)
> > 
> > Aren't highly complex, interdependent systems with different owners
> > of
> > different components fun? :-)
> 
> well, they pay my mortgage...:P

And some people's therapist's mortgage :-)

-- 
Doug Ledford 
GPG KeyID: B826A3330E572FDD
Fingerprint = AE6B 1BDA 122B 23B4 265B  1274 B826 A333 0E57 2FDD


signature.asc
Description: This is a digitally signed message part
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Lots of systemctl segfaults in Koji Rawhide

2020-05-19 Thread Adam Williamson 
On Tue, 2020-05-19 at 18:55 -0400, Doug Ledford wrote:
> 
> > I guess we get to poke through everything built around the 17th and
> > try
> > to find a relevant change? :)
> 
> Aren't highly complex, interdependent systems with different owners of
> different components fun? :-)

well, they pay my mortgage...:P
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Lots of systemctl segfaults in Koji Rawhide

2020-05-19 Thread Doug Ledford 
On Tue, 2020-05-19 at 15:45 -0700, Adam Williamson wrote:
> On Tue, 2020-05-19 at 18:16 -0400, Doug Ledford wrote:
> > But this is different, and it's the cause of your problem (well,
> > it's
> > the immediate cause anyway).  The kernel-install script is failing
> > because it's passing /etc/kernel/install.d/ to something that wants
> > something other than a directory.  That could be because a failed
> > scriptlet elsewhere has resulted in that directory being empty, so
> > some
> > glob is returning the directory instead of the files in the
> > directory,
> > so it could still be the systemctl issue, but it would be worth
> > looking
> > into install-kernel to see.
> > 
> > OK, so install-kernel is part of systemd (I'm seeing a pattern
> > here). 
> > It generates an array of plugins that should be called for the new
> > kernel.  It does all plugins in .install, /etc/kernel/install.d/,
> > and
> > /usr/lib/kernel/install.d.  My guess here is that the %POSTTRANS of
> > the
> > kernel-core package is calling install-kernel, which is failing with
> > the
> > above error, causing the kernel-core %POSTTRANS to error out
> > prematurely, resulting in the missing modules.dep file that is
> > breaking
> > your build.  At this point, I can't see what the problem can be
> > other
> > than either a bad build of systemd, or if the recent upgrade to
> > rdma-
> > core-29, with the new libibverbs-29 package, has caused a failure in
> > systemd because it needs relinked or something.  But that can only
> > be
> > the case if it's some sort of weak dependency based on dlopen as
> > both
> > the rpm tools and ldd are not picking up the libibverbs
> > dependency.  If
> > that's the case, the systemd and/or udev rpm packages should have an
> > explicit requires on libibverbs I think.
> > 
> > Anyway, at this point, I don't know if the rdma-core-owner people
> > can
> > help.  I think this is first in the hands of the systemd folks.
> 
> Well, systemd hasn't changed for nearly a month - it was last built on
> 2020-04-21. nbdkit was built successfully a few days ago, so blaming
> systemd here feels wrong.

Ok, then something else has changed that is causing systemd problems
then.

> It's notable that the last couple of Rawhide composes have also failed
> with an odd kernel-related error:
> 
> https://koji.fedoraproject.org/koji/taskinfo?taskID=44651583
> 
> DEBUG util.py:602:  Traceback (most recent call last):
> DEBUG util.py:602:File "/usr/sbin/lorax", line 223, in 
> DEBUG util.py:602:  main()
> DEBUG util.py:602:File "/usr/sbin/lorax", line 204, in main
> DEBUG util.py:602:  lorax.run(dnfbase, opts.product, opts.version,
> opts.release,
> DEBUG util.py:602:File "/usr/lib/python3.8/site-
> packages/pylorax/__init__.py", line 354, in run
> DEBUG
> util.py:602:  treebuilder.rebuild_initrds(add_args=anaconda_args)
> DEBUG util.py:602:File "/usr/lib/python3.8/site-
> packages/pylorax/treebuilder.py", line 308, in rebuild_initrds
> DEBUG util.py:602:  raise Exception("No kernels found, cannot
> rebuild_initrds")
> DEBUG util.py:602:  Exception: No kernels found, cannot
> rebuild_initrds
> 
> still, having trouble pinning down a culprit; it's not kernel-5.7.0-
> 0.rc6.1.fc33 as the 20200518.n.0 compose failed, and that was run with
> the previous kernel build, which *succeeded* in the 20200517.n.0
> compose...
> 
> I guess we get to poke through everything built around the 17th and
> try
> to find a relevant change? :)

Aren't highly complex, interdependent systems with different owners of
different components fun? :-)

-- 
Doug Ledford 
GPG KeyID: B826A3330E572FDD
Fingerprint = AE6B 1BDA 122B 23B4 265B  1274 B826 A333 0E57 2FDD


signature.asc
Description: This is a digitally signed message part
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Lots of systemctl segfaults in Koji Rawhide

2020-05-19 Thread Adam Williamson 
On Tue, 2020-05-19 at 18:16 -0400, Doug Ledford wrote:
> 
> But this is different, and it's the cause of your problem (well, it's
> the immediate cause anyway).  The kernel-install script is failing
> because it's passing /etc/kernel/install.d/ to something that wants
> something other than a directory.  That could be because a failed
> scriptlet elsewhere has resulted in that directory being empty, so some
> glob is returning the directory instead of the files in the directory,
> so it could still be the systemctl issue, but it would be worth looking
> into install-kernel to see.
> 
> OK, so install-kernel is part of systemd (I'm seeing a pattern here). 
> It generates an array of plugins that should be called for the new
> kernel.  It does all plugins in .install, /etc/kernel/install.d/, and
> /usr/lib/kernel/install.d.  My guess here is that the %POSTTRANS of the
> kernel-core package is calling install-kernel, which is failing with the
> above error, causing the kernel-core %POSTTRANS to error out
> prematurely, resulting in the missing modules.dep file that is breaking
> your build.  At this point, I can't see what the problem can be other
> than either a bad build of systemd, or if the recent upgrade to rdma-
> core-29, with the new libibverbs-29 package, has caused a failure in
> systemd because it needs relinked or something.  But that can only be
> the case if it's some sort of weak dependency based on dlopen as both
> the rpm tools and ldd are not picking up the libibverbs dependency.  If
> that's the case, the systemd and/or udev rpm packages should have an
> explicit requires on libibverbs I think.
> 
> Anyway, at this point, I don't know if the rdma-core-owner people can
> help.  I think this is first in the hands of the systemd folks.

Well, systemd hasn't changed for nearly a month - it was last built on
2020-04-21. nbdkit was built successfully a few days ago, so blaming
systemd here feels wrong.

It's notable that the last couple of Rawhide composes have also failed
with an odd kernel-related error:

https://koji.fedoraproject.org/koji/taskinfo?taskID=44651583

DEBUG util.py:602:  Traceback (most recent call last):
DEBUG util.py:602:File "/usr/sbin/lorax", line 223, in 
DEBUG util.py:602:  main()
DEBUG util.py:602:File "/usr/sbin/lorax", line 204, in main
DEBUG util.py:602:  lorax.run(dnfbase, opts.product, opts.version, 
opts.release,
DEBUG util.py:602:File 
"/usr/lib/python3.8/site-packages/pylorax/__init__.py", line 354, in run
DEBUG util.py:602:  treebuilder.rebuild_initrds(add_args=anaconda_args)
DEBUG util.py:602:File 
"/usr/lib/python3.8/site-packages/pylorax/treebuilder.py", line 308, in 
rebuild_initrds
DEBUG util.py:602:  raise Exception("No kernels found, cannot 
rebuild_initrds")
DEBUG util.py:602:  Exception: No kernels found, cannot rebuild_initrds

still, having trouble pinning down a culprit; it's not kernel-5.7.0-
0.rc6.1.fc33 as the 20200518.n.0 compose failed, and that was run with
the previous kernel build, which *succeeded* in the 20200517.n.0
compose...

I guess we get to poke through everything built around the 17th and try
to find a relevant change? :)
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Lots of systemctl segfaults in Koji Rawhide

2020-05-19 Thread Doug Ledford 
On Tue, 2020-05-19 at 19:02 +0100, Richard W.M. Jones wrote:
> I was trying to chase down why modules.dep is no longer built in
> Rawhide (which affects supermin and therefore libguestfs), but it
> looks like there's a much more serious problem:
> 
> https://kojipkgs.fedoraproject.org//work/tasks/8242/44698242/root.log
> (from https://koji.fedoraproject.org/koji/taskinfo?taskID=44698142)
> 
> DEBUG util.py:602:  Running transaction
> DEBUG util.py:602:  /usr/bin/systemctl: error while loading shared
> libraries: libibverbs.so.1: cannot open shared object file: No such
> file or directory
> DEBUG util.py:602:  /sbin/udevadm: error while loading shared
> libraries: libibverbs.so.1: cannot open shared object file: No such
> file or directory
> DEBUG util.py:602:  /sbin/udevadm: error while loading shared
> libraries: libibverbs.so.1: cannot open shared object file: No such
> file or directory
> DEBUG util.py:602:  /sbin/udevadm: error while loading shared
> libraries: libibverbs.so.1: cannot open shared object file: No such
> file or directory

Just to double check, I installed a rawhide installation and then ran
ldd on both systemctl and udevadm.  Neither of these are linked to
libibverbs as far as I can tell.  If they use libibverbs, they do it via
dlopen, not static linking.  So that being the case, these errors make
no sense.

> DEBUG util.py:602:  /var/tmp/rpm-tmp.pYK3f6: line 6: 2005996
> Segmentation fault  (core dumped) /usr/bin/systemctl --no-reload
> preset dbus.socket

Starting here, the error changes to a core dump instead of a library not
found, so maybe this is after the libibverbs package is finally
installed and libibverbs.so.1 is now found?

> DEBUG util.py:602:  /var/tmp/rpm-tmp.pYK3f6: line 13: 2005999
> Segmentation fault  (core dumped) /usr/bin/systemctl --no-reload
> preset \--global dbus.socket
> DEBUG util.py:602:  /var/tmp/rpm-tmp.heiSf7: line 6: 2006029
> Segmentation fault  (core dumped) /usr/bin/systemctl --no-reload
> preset dbus-broker.service
> DEBUG util.py:602:  /var/tmp/rpm-tmp.heiSf7: line 13: 2006032
> Segmentation fault  (core dumped) /usr/bin/systemctl --no-reload
> preset \--global dbus-broker.service
> DEBUG util.py:602:  /var/tmp/rpm-tmp.NScHA9: line 9: 2006061
> Segmentation fault  (core dumped) /usr/bin/systemctl --no-reload
> preset polkit.service
> DEBUG util.py:602:  /var/tmp/rpm-tmp.05AcI8: line 6: 2006069
> Segmentation fault  (core dumped) /usr/bin/systemctl --no-reload
> preset dm-event.socket
> DEBUG util.py:602:  /var/tmp/rpm-tmp.05AcI8: line 11: 2006072
> Segmentation fault  (core dumped) systemctl enable dm-event.socket
> DEBUG util.py:602:  /var/tmp/rpm-tmp.05AcI8: line 12: 2006080
> Segmentation fault  (core dumped) systemctl start dm-event.socket
> > /dev/null 2>&1
> DEBUG util.py:602:  /var/tmp/rpm-tmp.4iXDa7: line 6: 2006113
> Segmentation fault  (core dumped) /usr/bin/systemctl --no-reload
> preset saslauthd.service
> DEBUG util.py:602:  /var/tmp/rpm-tmp.8CYlZ7: line 6: 2006121
> Segmentation fault  (core dumped) /usr/bin/systemctl --no-reload
> preset rdisc.service
> DEBUG util.py:602:  /var/tmp/rpm-tmp.mIKYA7: line 6: 2006130
> Segmentation fault  (core dumped) /usr/bin/systemctl --no-reload
> preset mdmonitor.service raid-check.{service,timer}
> DEBUG util.py:602:  /var/tmp/rpm-tmp.mIKYA7: line 8: 2006133
> Segmentation fault  (core dumped) /usr/bin/systemctl disable
> mdmonitor-takeover.service > /dev/null 2>&1
> DEBUG util.py:602:  /var/tmp/rpm-tmp.pk6AR5: line 6: 2006141
> Segmentation fault  (core dumped) /usr/bin/systemctl --no-reload
> preset numad.service
> DEBUG util.py:602:  /var/tmp/rpm-tmp.eLxfL5: line 7: 2006179
> Segmentation fault  (core dumped) /usr/bin/systemctl --no-reload
> preset virtlockd.socket virtlockd-admin.socket
> DEBUG util.py:602:  /var/tmp/rpm-tmp.eLxfL5: line 14: 2006182
> Segmentation fault  (core dumped) /usr/bin/systemctl --no-reload
> preset virtlogd.socket virtlogd-admin.socket
> DEBUG util.py:602:  /var/tmp/rpm-tmp.eLxfL5: line 21: 2006189
> Segmentation fault  (core dumped) /usr/bin/systemctl --no-reload
> preset libvirtd.socket libvirtd-ro.socket libvirtd-admin.socket
> DEBUG util.py:602:  /var/tmp/rpm-tmp.eLxfL5: line 28: 2006196
> Segmentation fault  (core dumped) /usr/bin/systemctl --no-reload
> preset libvirtd-tcp.socket libvirtd-tls.socket
> DEBUG util.py:602:  /var/tmp/rpm-tmp.eLxfL5: line 35: 2006203
> Segmentation fault  (core dumped) /usr/bin/systemctl --no-reload
> preset libvirtd.service
> DEBUG util.py:602:  /var/tmp/rpm-tmp.bk1ZR5: line 6: 2006222
> Segmentation fault  (core dumped) /usr/bin/systemctl --no-reload
> preset blk-availability.service lvm2-monitor.service
> DEBUG util.py:602:  /var/tmp/rpm-tmp.bk1ZR5: line 13: 2006225
> Segmentation fault  (core dumped) systemctl enable lvm2-
> monitor.service
> DEBUG util.py:602:  /var/tmp/rpm-tmp.bk1ZR5: line 13: 2006232
> Segmentation fault  (core dumped) systemctl

Lots of systemctl segfaults in Koji Rawhide

2020-05-19 Thread Richard W.M. Jones 
I was trying to chase down why modules.dep is no longer built in
Rawhide (which affects supermin and therefore libguestfs), but it
looks like there's a much more serious problem:

https://kojipkgs.fedoraproject.org//work/tasks/8242/44698242/root.log
(from https://koji.fedoraproject.org/koji/taskinfo?taskID=44698142)

DEBUG util.py:602:  Running transaction
DEBUG util.py:602:  /usr/bin/systemctl: error while loading shared libraries: 
libibverbs.so.1: cannot open shared object file: No such file or directory
DEBUG util.py:602:  /sbin/udevadm: error while loading shared libraries: 
libibverbs.so.1: cannot open shared object file: No such file or directory
DEBUG util.py:602:  /sbin/udevadm: error while loading shared libraries: 
libibverbs.so.1: cannot open shared object file: No such file or directory
DEBUG util.py:602:  /sbin/udevadm: error while loading shared libraries: 
libibverbs.so.1: cannot open shared object file: No such file or directory
DEBUG util.py:602:  /var/tmp/rpm-tmp.pYK3f6: line 6: 2005996 Segmentation fault 
 (core dumped) /usr/bin/systemctl --no-reload preset dbus.socket
DEBUG util.py:602:  /var/tmp/rpm-tmp.pYK3f6: line 13: 2005999 Segmentation 
fault  (core dumped) /usr/bin/systemctl --no-reload preset \--global 
dbus.socket
DEBUG util.py:602:  /var/tmp/rpm-tmp.heiSf7: line 6: 2006029 Segmentation fault 
 (core dumped) /usr/bin/systemctl --no-reload preset dbus-broker.service
DEBUG util.py:602:  /var/tmp/rpm-tmp.heiSf7: line 13: 2006032 Segmentation 
fault  (core dumped) /usr/bin/systemctl --no-reload preset \--global 
dbus-broker.service
DEBUG util.py:602:  /var/tmp/rpm-tmp.NScHA9: line 9: 2006061 Segmentation fault 
 (core dumped) /usr/bin/systemctl --no-reload preset polkit.service
DEBUG util.py:602:  /var/tmp/rpm-tmp.05AcI8: line 6: 2006069 Segmentation fault 
 (core dumped) /usr/bin/systemctl --no-reload preset dm-event.socket
DEBUG util.py:602:  /var/tmp/rpm-tmp.05AcI8: line 11: 2006072 Segmentation 
fault  (core dumped) systemctl enable dm-event.socket
DEBUG util.py:602:  /var/tmp/rpm-tmp.05AcI8: line 12: 2006080 Segmentation 
fault  (core dumped) systemctl start dm-event.socket > /dev/null 2>&1
DEBUG util.py:602:  /var/tmp/rpm-tmp.4iXDa7: line 6: 2006113 Segmentation fault 
 (core dumped) /usr/bin/systemctl --no-reload preset saslauthd.service
DEBUG util.py:602:  /var/tmp/rpm-tmp.8CYlZ7: line 6: 2006121 Segmentation fault 
 (core dumped) /usr/bin/systemctl --no-reload preset rdisc.service
DEBUG util.py:602:  /var/tmp/rpm-tmp.mIKYA7: line 6: 2006130 Segmentation fault 
 (core dumped) /usr/bin/systemctl --no-reload preset mdmonitor.service 
raid-check.{service,timer}
DEBUG util.py:602:  /var/tmp/rpm-tmp.mIKYA7: line 8: 2006133 Segmentation fault 
 (core dumped) /usr/bin/systemctl disable mdmonitor-takeover.service > 
/dev/null 2>&1
DEBUG util.py:602:  /var/tmp/rpm-tmp.pk6AR5: line 6: 2006141 Segmentation fault 
 (core dumped) /usr/bin/systemctl --no-reload preset numad.service
DEBUG util.py:602:  /var/tmp/rpm-tmp.eLxfL5: line 7: 2006179 Segmentation fault 
 (core dumped) /usr/bin/systemctl --no-reload preset virtlockd.socket 
virtlockd-admin.socket
DEBUG util.py:602:  /var/tmp/rpm-tmp.eLxfL5: line 14: 2006182 Segmentation 
fault  (core dumped) /usr/bin/systemctl --no-reload preset virtlogd.socket 
virtlogd-admin.socket
DEBUG util.py:602:  /var/tmp/rpm-tmp.eLxfL5: line 21: 2006189 Segmentation 
fault  (core dumped) /usr/bin/systemctl --no-reload preset libvirtd.socket 
libvirtd-ro.socket libvirtd-admin.socket
DEBUG util.py:602:  /var/tmp/rpm-tmp.eLxfL5: line 28: 2006196 Segmentation 
fault  (core dumped) /usr/bin/systemctl --no-reload preset 
libvirtd-tcp.socket libvirtd-tls.socket
DEBUG util.py:602:  /var/tmp/rpm-tmp.eLxfL5: line 35: 2006203 Segmentation 
fault  (core dumped) /usr/bin/systemctl --no-reload preset libvirtd.service
DEBUG util.py:602:  /var/tmp/rpm-tmp.bk1ZR5: line 6: 2006222 Segmentation fault 
 (core dumped) /usr/bin/systemctl --no-reload preset 
blk-availability.service lvm2-monitor.service
DEBUG util.py:602:  /var/tmp/rpm-tmp.bk1ZR5: line 13: 2006225 Segmentation 
fault  (core dumped) systemctl enable lvm2-monitor.service
DEBUG util.py:602:  /var/tmp/rpm-tmp.bk1ZR5: line 13: 2006232 Segmentation 
fault  (core dumped) systemctl start lvm2-monitor.service > /dev/null 2>&1
DEBUG util.py:602:  /var/tmp/rpm-tmp.bk1ZR5: line 20: 2006239 Segmentation 
fault  (core dumped) /usr/bin/systemctl --no-reload preset 
lvm2-lvmpolld.socket
DEBUG util.py:602:  /var/tmp/rpm-tmp.bk1ZR5: line 25: 2006246 Segmentation 
fault  (core dumped) systemctl enable lvm2-lvmpolld.socket
DEBUG util.py:602:  /var/tmp/rpm-tmp.bk1ZR5: line 26: 2006254 Segmentation 
fault  (core dumped) systemctl start lvm2-lvmpolld.socket > /dev/null 2>&1
DEBUG util.py:602:  /var/tmp/rpm-tmp.M5VjN9: line 6: 2006725 Segmentation fault 
 (core dumped) /usr/bin/systemctl --no-reload preset brltty.service
DEBUG